Introduction to Shell Scripting

In Text Streams and Redirection, I went over some helpful CLI tools and shell operators to manage text streams. It’s all well and good to see examples of grep and tr, but it doesn’t give a full sense of how powerful the shell environment actually is.

I recently wrote a script to help me automate a very tedious task, and I thought it would be useful to break it down here. If you follow me on Twitter, you’re aware that I’ve been actively exploring and implementing IPFS to store NFT data for the Degenimals collection on OpenSea.

I will write about IPFS and NFTs at a later date, but it helps to serve as a backdrop for why I wrote the script.

About Shell Scripts

Linux shells are capable of very powerful scripting patterns, and they include a lot of common logic used in more complicated programming languages. Among these:

  • Functions
  • Loops
  • Conditional statements
  • Variables
  • Inline string modification
  • Strings, arrays, dictionaries
  • Basic calculations

… and more!

I’m not going to get deep in the weeds with how to write shell scripts, but take my word that it’s quite powerful.

The OpenSea Script

The goal for the script was to download the avatar images associated with the minted Degenimals. I wanted to have a record copy of all avatar images, as well as their official NFT names, for easy publishing over IPFS.

The obvious and tedious solution is to visit the OpenSea collection and download them all by hand. But where’s the fun in that? Anything done by hand introduces the chance for human error.

Luckily for us, OpenSea has a documented API that we can use. The API is accessible over HTTP, which means that I can use CLI tools and shell scripting to handle it all for me.

Getting Started

The OpenSea API returns data in JSON format. It’s not super relevant to the shell scripting discussion, but knowing the output is critical so we can select the appropriate tool to process it. In our case, we use jq to process our received JSON data.

The first thing we need to do is retrieve some data from the API and see what we get.

First, let’s discuss how to retrieve HTTP data on the CLI. The tool you want is curl, which simply sends HTTP calls and sends the resulting HTTP data to your shell. You can test it by executing curl http://google.com (or similar) and seeing the resulting HTML.

My initial attempts at getting good data from OpenSea were frustrating, because the default call returns 20 assets at a time. I’m not skilled enough with jq to sift through a JSON array with multiple assets, so I edited the API call to request a single result. Here is that command:

devil@ubuntuVM:~$ curl -s "https://api.opensea.io/api/v1/assets?order_direction=desc&collection=degenimals&offset=0&limit=1"
{"assets":[{"id":30740264,"token_id":"47475974381565896501680608588151947420114181955589068365149607303160971395073","num_sales":0,"background_color":null,"image_url":"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw","image_preview_url":"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw=s250","image_thumbnail_url":"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw=s128","image_original_url":null,"animation_url":null,"animation_original_url":null,"name":"BowTiedHeifer","description":"The Official BowTiedHeifer\n\nMember of The @BowTiedJungle \n\nOwner of @BowTiedHeifer Twitter Account\n\nWelcome, anon","external_link":null,"asset_contract":{"address":"0x495f947276749ce646f68ac8c248420045cb7b5e","asset_contract_type":"semi-fungible","created_date":"2020-12-02T17:40:53.232025","name":"OpenSea Collection","nft_version":null,"opensea_version":"2.0.0","owner":102384,"schema_name":"ERC1155","symbol":"OPENSTORE","total_supply":null,"description":null,"external_link":null,"image_url":null,"default_to_fiat":false,"dev_buyer_fee_basis_points":0,"dev_seller_fee_basis_points":0,"only_proxied_transfers":false,"opensea_buyer_fee_basis_points":0,"opensea_seller_fee_basis_points":250,"buyer_fee_basis_points":0,"seller_fee_basis_points":250,"payout_address":null},"permalink":"https://opensea.io/assets/0x495f947276749ce646f68ac8c248420045cb7b5e/47475974381565896501680608588151947420114181955589068365149607303160971395073","collection":{"banner_image_url":"https://lh3.googleusercontent.com/2keCX7dNnAObsnKCi06-4djU4L8W8x_O_OqnRsXJbbP7yQR-lBaSxQTjwBlcPzThBvslU_yf5h0sqNzI55qS1MoKSuTU7IDJJcCS3Q=s2500","chat_url":null,"created_date":"2021-05-11T19:30:15.242986","default_to_fiat":false,"description":"The Official BowTiedJungle NFT Collection\n\nClaim & Mint your Official Jungle Avatar. Influenced & Inspired by BowTiedBull.\n\nWelcome, anon","dev_buyer_fee_basis_points":"0","dev_seller_fee_basis_points":"800","discord_url":"https://discord.gg/K6WfHphzBj","display_data":{"card_display_style":"padded"},"external_url":null,"featured":false,"featured_image_url":"https://lh3.googleusercontent.com/1Wj-MOUHtt3KiFXqmVtPY727PTEodWQWjFUCEeMQVYpujbqlsPSSV4E2K0AfnbGLR0gWbadQE5L0sU_maLp0MibI0X9qDW7z7QY2kPQ=s300","hidden":false,"safelist_request_status":"not_requested","image_url":"https://lh3.googleusercontent.com/kioJLaD1IOZadm-2zIc-WFZWFLWfdemun5ZTeb3tCAmSn6nBGTw3YEvS1pfoYHvJzHdxi87LQI5yRgAICMioFpMGkoc360p9s-mSfc0=s120","is_subject_to_whitelist":false,"large_image_url":"https://lh3.googleusercontent.com/1Wj-MOUHtt3KiFXqmVtPY727PTEodWQWjFUCEeMQVYpujbqlsPSSV4E2K0AfnbGLR0gWbadQE5L0sU_maLp0MibI0X9qDW7z7QY2kPQ=s300","medium_username":null,"name":"Degenimals","only_proxied_transfers":false,"opensea_buyer_fee_basis_points":"0","opensea_seller_fee_basis_points":"250","payout_address":"0x68f67301dc1d9a46211eda5c0408de7e44527423","require_email":false,"short_description":null,"slug":"degenimals","telegram_url":null,"twitter_username":"https://twitter.com/Degenimals?s=20","instagram_username":null,"wiki_url":null},"decimals":null,"token_metadata":null,"owner":{"user":{"username":"NullAddress"},"profile_img_url":"https://storage.googleapis.com/opensea-static/opensea-profile/1.png","address":"0x0000000000000000000000000000000000000000","config":"","discord_id":""},"sell_orders":null,"creator":{"user":{"username":"BowTiedRafiki"},"profile_img_url":"https://storage.googleapis.com/opensea-static/opensea-profile/33.png","address":"0x68f67301dc1d9a46211eda5c0408de7e44527423","config":"","discord_id":""},"traits":[{"trait_type":"Animal Group","value":"Mammal","display_type":null,"max_value":null,"trait_count":101,"order":null},{"trait_type":"Mammal","value":"Artiodactyla","display_type":null,"max_value":null,"trait_count":13,"order":null},{"trait_type":"Artiodactyla","value":"Cow","display_type":null,"max_value":null,"trait_count":2,"order":null},{"trait_type":"Degenimal ID","value":149,"display_type":"number","max_value":"150","trait_count":0,"order":null}],"last_sale":null,"top_bid":null,"listing_date":null,"is_presale":true,"transfer_fee_payment_token":null,"transfer_fee":null}]}

The key is the offset and limit parameters in the API call. By specifying a single item with offset and limiting the result to 1, I could retrieve the JSON data for a single Degenimal.

Filtering the Data

jq saves the day here by allowing us to filter for a particular item. In this case, the part I want is the image_url, which represents the avatar image stored on a Google server. You can look through it above and see that in this case, BowTiedHeifer’s image was stored at "https://lh3.googleusercontent.com/kioJLaD1IOZadm-2zIc-WFZWFLWfdemun5ZTeb3tCAmSn6nBGTw3YEvS1pfoYHvJzHdxi87LQI5yRgAICMioFpMGkoc360p9s-mSfc0=s120"

Rather than picking through this by hand, jq allows me to search for it by piping (remember that?) the output of curl into it and filtering for an element called image_url inside the assets array.

devil@ubuntuVM:~$ curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=0&limit=1"| jq '.assets[].image_url'
"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw"

Nice, that worked! Now I have the image_url for the first Degenimal at offset=0 (in descending order).

OK, but the image_url has no indication that it belongs to BowTiedHeifer. I need some way to label the asset later, so let’s adjust our jq call to find the name instead of the image_url.

devil@ubuntuVM:~$ curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=0&limit=1"| jq '.assets[].name'
"BowTiedHeifer"

Storing the Asset

Excellent! Now we have a way to find the Degenimal name and get an URL for the image. Now we can get into using variables.

The format for a shell script variable is var=$(some command)

Let’s use name as our variable for the Degenimal, and then set it equal to the command we just discovered above.

devil@ubuntuVM:~$ name=$(curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=$i&limit=1"| jq '.assets[].name')
devil@ubuntuVM:~$ echo $name
"BowTiedHeifer"

Hmm… Looks good except that I don’t want those double quotes, so I edit the command by piping it into tr (remember that?) to delete all instances of ".

devil@ubuntuVM:~$ name=$(curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=$i&limit=1"| jq '.assets[].name' | tr -d '"')
devil@ubuntuVM:~$ echo $name
BowTiedHeifer

There we go!

Now I have some easily-reused commands to retrieve JSON data from OpenSea, filter for name, and filter for the image_url.

To retrieve something over HTTP, I use the wget command. It is called in the form wget [URL] to output to the shell, or wget [URL] -O [filename] to save to a file. Since I’m trying to download images, I use the -O option.

I already have image_url and name, so I simply send those variables into my wget command.

devil@ubuntuVM:~$ wget $image\_url -O assets/image/$name
Connecting to lh3.googleusercontent.com ([2607:f8b0:4005:807::2001]:443)
saving to 'BowTiedHeifer'
BowTiedHeifer        100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*| 50271  0:00:00 ETA
'BowTiedHeifer' saved

That’s a little noisy, so I check the wget man page to find the -q option (quiet output).

My command now looks like wget -q $image_url -O assets/image/$name.

Looping Through All Degenimals

Now that I have a nice set of commands to retrieve JSON data, filter for name and image_url, and save the file using wget, I’m ready to apply this to the whole set of Degenimals.

I can use the for i in ... pattern to construct a loop. Since the API returns assets starting at 0, and the maximum size of the Degenimal collection is 150, I build my loop: for i in $(seq 0 149). This will loop through all the Degenimals assets starting from the top and going until it’s reached 150 (I started from 0 so I only count to 149).

Then I re-write my data=$(curl -s "https://api.opensea.io/api/v1/assets?order_direction=desc&collection=degenimals&offset=0&limit=1" | jq ) command to use my new counting variable i instead of 0. In this way, the command will be re-written each time the for loop executes. It will start at 0, then go to 1, then go to 2, etc.

All Together

#!/bin/sh

mkdir -p assets/image

for i in $(seq 0 149)
do
        # get JSON data from OpenSea API
        data=$(curl -s "https://api.opensea.io/api/v1/assets?order_direction=desc&collection=degenimals&offset=$i&limit=1" | jq )

        # get Degenimal name and image_url from retrieved data
        name=$(echo $data | jq '.assets[].name' | tr -d '"')
        image_url=$(echo $data | jq '.assets[].image_url' | tr -d '"')

        echo -n "Processing Degenimal ($name)... "

        # if assets already exist, skip
        if test -f ./assets/image/$name
        then
                echo "asset exists, skipped"
        else
                echo -n "downloading... "
                wget -q $image_url -O assets/image/$name && echo "done"
        fi

        # sleep to avoid hitting API rate limit
        sleep 1
done

Running the script results in the following output:

devil@ubuntuVM:~$ sh ./opensea\_api\_process.sh
Processing Degenimal (BowTiedHeifer)... downloading... done
Processing Degenimal (BowTiedFarmer)... downloading... done
Processing Degenimal (BowTiedWestie)... downloading... done
Processing Degenimal (BowTiedSable)... downloading... done
Processing Degenimal (BowTiedSnipe)... downloading... done
Processing Degenimal (BowTiedNarwhal)... downloading... done
[output clipped]

Bang, done! A few minutes monkeying on the CLI and I’ve written a script that will save me hours of work.

Wrapping Up

Admittedly this post was a bit intense, so don’t worry if this went over your head. If you don’t care to play along with this one, please consider it a useful example of how many powerful things the Linux CLI allows you to do.

Could you do something like this on Windows? Mac? Your Phone?

Newsletter

Tip Jar

If you're getting value from my writing, please support my efforts with a donation. You can donate directly using my public Ethereum address bowtieddevil.eth. Or you can use the donation button below, which works through my self-hosted BTCPay Server.
linux 

See also