Scripting

Scripting is a programming language which does not need in a compilation, they interpreted. Scripting using almost everywhere such as web pages, the shell of operating systems, software applications, games. It’s really common to use javascript for handling user click on the button.

Currently, we have a bunch of scripting languages such as javascript, python, Perl, shell script and many others. JavaScript basically used in programming web pages, Python for building a wide range of scripts, bash shell script for building script for OS(file manipulation, program execution).

Problem

Sometimes one technology does not have some features like in other technologies. For example, bash shell script does not have default features for JSON manipulation, when other technologies like JavaScript have this features by default. And it looks natural when we use technologies for the intended purpose and combine multiple technologies.

Also, we all are human, and we have a memory, and our memory can miss some information, for example, create a file in JS, but we clearly know how to create a file by using a bash touch.

For example, we are a JS developer, and we know how to use map/reduce/filter function, but we do not know how to make it through the bash. Firstly we start googling how to use map function in bash, instead of use JavaScript in a terminal. If you select a Zpy you can combine multiple script languages in 1 command line shell.

What is Zpy?

Zpy is command line shell where you can use not only 1 language but combine multiple languages and send data between them. You can create an HTTP request by using wget, parse data by using a Python language and preprocess data by using JavaScript.

Hello world

Let’s start with something really simple, for example just printing “Hello world” and make some arithmetical operations.

(Zpy) "Hello world"
Hello world
(Zpy) 4 + 7
11
(Zpy) (5 + 3 * 8)/5
5.8

By default Zpy, use a Python language. But we can use bash language if we will add symbol ` at begin or j if we want to use a JavaScript. For import module in Python, we need to add symbol ~ at begin which indicate a Python language.

(Zpy) ~import math, random
(Zpy) math.sin(math.e * random.random())
0.7826813558152167
(Zpy) math.sin(math.e * random.random())
0.4710093839817176

For importing module to javascript, we need use require keyword, and install the package globally by using npm.

Creation files with random names

Let’s imagine about the real task - Create 5 empty files with random names.

(Zpy) j new Array(5)
	|j z.map((e)=>Math.random())
	|j z.join(" ")
	| touch $z
(Zpy) ls
0.04070692726231173	0.604588968489302
0.51512968378711	0.9823757457193931
0.5302334711282355

Here we use a pipe symbol | for interacting between languages. Symboj j means that we use an JavaScript language, for bash language we use a ` symbol, and Python language using by default.

Symbol z is an input argument which will be sent from the previous block by pipeline, in the second block we receive an empty array of 5 which will be sent from the first block into variable z.

Here we have an 4 blocks:

  • j new Array(5) - creating an empty array of size 5
  • j z.map((e)=>Math.random()) - here we receiving empty array, and use map function for generate array of random numbers.
  • j z.join(" ") - joining array of random digits by using ` ` symbol.
  • touch $z - creating files when filenames is an random numbers from previous block.

As we can see, this task will be an easily solved, if we will use a Zpy.

Simple http static server

Let’s try to create a simple static server by using connect library. On the first step, we installed globally connect package and serve-static package. After that, we import these modules, and also import random module to python. And at the end, we create a random folder in /tmp folder, fill index.html file and run the static server on 8081 port.

(Zpy) ` npm i connect serve-static -G
(Zpy) j connect = require('connect')
Added new requirements : { [['connect', 'connect']] }
(Zpy) j serveStatic = require('serve-static')
Added new requirements : { [['serveStatic', 'serve-static']] }
(Zpy) ~import random
(Zpy) random.random()  # Generate random number
	| "/tmp/server_%s" % z  # Generate string
	|` mkdir $z; echo $z  # Create server folder and echo location
	| "%s/index.html" % z.strip() # Generate filename string
	| echo "Hello world from index.html which located at ${z}" > $z; echo $z # Fill index.html and return location of file
	| j connect().use(serveStatic(  require('path').dirname(z)) ).listen(8081) #Create static server

static server If you look at the third line, we create a folder and print variable mkdir $z; echo $z, if we want to send the value to the next block, we need to return value which will be sent to the next block.

Http request

Let’s try to do something more complicated, like a http request. Here we can use wget or curl, but what we should do if we need extract some data from http response and this response was be encoded in JSON for example?

The first link in google says we can use a JSON preprocessor jq. After installing on OSX and run some simple request, it works great, but if we need to do something complicated, we should read a documentation how to use this library, instead of using familiar languages like Python or JS for solving this task.

> brew install jq
> curl -s 'https://api.github.com/users/lambda' | jq -r '.name'
Brian Campbell

Json preprocessor jq

Here we can use a Zpy for the solve this simple task, I will use a JavaScript JSON.parse function and Python for preprocessing data.

(Zpy) `curl -s 'https://api.github.com/users/lambda' |j JSON.parse(z).name
Brian Campbell
(Zpy) `curl -s 'https://api.github.com/users/lambda' |j JSON.par
se(z)| "%s - %s" % (z['type'], z['name'])
User - Brian Campbell

Using Zpy for preprocess json request

And for sure we can send this data as http request, to requestb.in. Note: Firstly you need to create an endpoint on requestb.in, just visit this service and create

` curl -s 'https://api.github.com/users/lambda' 
	|j JSON.parse(z)| "%s - %s" % (z['type'], z['name']) 
	|` curl -d "data=$z" -X POST https://requestb.in/1gpnzpl1

Sending response data from github api to requestb.in as http request

Chain pool [for] function

[for] function is a really useful in Zpy. [for] function iterate through every item in array or data split by \n character as stdin and evaluate every iterated item by any available other languages in Zpy such as JS, Python or bash.

For example, we need to list a directory of multiple folders. It’s really easy to do this by using [for] function.

(Zpy) ['folder1', 'folder2'] 
	| [for] ls $z 
subfolder1_1	subfolder1_2
subfolder2_1	subfolder2_2	subfolder2_3	subfolder2_test

for function

You can read more about chain pool functions here.

Javascript async

It, not a secret that async function is the main part of javascript, for example, async Programming uses in the heart of Node.js - event loop. In Zpy we can convert async function to not async, simply just call a callback function sync.

(Zpy) j setTimeout(() => sync(13), 1100)
13

Return result from async function

Let’s try to retrieve posts from REST API endpoint, we just need to change post id in HTTP query, for example on this endpoint https://jsonplaceholder.typicode.com/posts?id=1, here need to use chain pool function for for create multiple http requests.

(Zpy) list(range(1,10)) 
	|[for] j request('https://jsonplaceholder.typicode.com/posts?id='+z, (err, res, body) => sync(JSON.parse(body)[0].title) )
['sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'qui est esse', 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'eum et est occaecati', 'nesciunt quas odio', 'dolorem eum magni eos aperiam quia', 'magnam facilis autem', 'dolorem dolore est ipsam', 'nesciunt iure omnis dolorem tempora et accusantium']

Send multiple http requests

Here we generate an array of ids by Python, after that, we use [for] keyword for creating multiple http requests. This requests moved into the queue, and Zpy evaluates each of task in a single thread.

Examples from real life

Simple Zpy script for moving screenshots from desktop to another folder.

By default, if you create a screenshot on OSX, it will be saved in the desktop folder, let’s try to create a script which moves all screenshots from desktop to another folder.

You can write zpy scripts on file, and run then by send location of file into zpy program as input argument.

> echo "j new Array(5)|j z.map((e)=>Math.random())"> script.zpy
> cat script.zpy
> j new Array(5)|j z.map((e)=>Math.random())
> zpy script.zpy
[0.889856010190299, 0.02441660903336995, 0.23200594971405852, 0.47496638325725793, 0.5552667771238191]

script.zpy - a simple zpy script

Let’s write a simple script which moves all files which contain “Screen Shot” in the filename, to the screenshot folder.

#!/Library/Frameworks/Python.framework/Versions/3.5/bin/zpy
ls ~/Desktop 
	| grep "Screen Shot" 
	| z.split("\n")
	|j z.filter(e => e.length > 0).map(e => "/Users/albert/Desktop/" + e) 
	|[for] mv "${z}" ~/Desktop/screenshot

desktop.zpy script, which move screenshot from desktop to another folder

After that we can easely add this script to crontab.

Ticket buying problem

Once I need to buy ticket, but ticket platform doesnt have available tickets, so I need to monitor website by continuously reloading the page. After 5 minute I decide automatize this task by using Zpy. I spend 5 minutes to write script and bought a ticket in 1 hour.

The first step I discover an HTTP request which sended to platform API. API endpoint returned an object where field value is an array of available trains. I was looking for new avaible trains so I need to be notified when new train tickets is to be avaible. I need to finish selected steps to create a simple notification system for ticket buying:

  • Create HTTP request
  • Parse JSON
  • Notify by using a beep sound if number of available trains greater that currenty available, in my case is 1.

For creating an HTTP request I need just right-click on request and copy request as cURL. For parsing json, I will use a JS and for sending the notification just print \a symbol.

ticket request Copy HTTP request as cURL

Final script will look like:

#!/Library/Frameworks/Python.framework/Versions/3.5/bin/zpy
` curl 'http://booking.uz.gov.ua/en/purchase/search/' -H 'GV-Ajax: 1' -H 'Origin: http://booking.uz.gov.ua' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8,ru;q=0.6,uk;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: */*' -H 'Referer: http://booking.uz.gov.ua/en/' -H 'Cookie:  HTTPSERVERID=server2; _gv_sessid=ifrpd9gnvrsmc90k8k99tihh32; _ga=GA1.3.1694846096.1499675049; _gid=GA1.3.200677417.1502873273; _gat=1; _gv_lang=en' -H 'Connection: keep-alive' -H 'GV-Screen: 1440x900' -H 'GV-Referer: http://booking.uz.gov.ua/en/' --data 'station_id_from=2204001&station_id_till=2208001&station_from=Kharkiv&station_till=Odesa&date_dep=08.31.2017&time_dep=00%3A00&time_dep_till=&another_ec=0&search=' --compressed
	|j JSON.parse(z).value 
	| print("\a" * 20) if len(z) > 1 else print("no")

ticket.zpy

But this script will be evaluated just once, so we need to write a simple program which will run this script multiple times in loop with random delays, I used a Python.

import random, time
from subprocess import call
while(True):
	time.sleep(5 + random.randint(0,10))
	call(['zpy','ticket.zpy'])

Simple python script which run our Zpy script for ticket availability