In Text Streams and Redirection, I went over some helpful CLI tools and shell operators to manage text streams. It’s all well and good to see examples of grep and tr, but it doesn’t give a full sense of how powerful the shell environment actually is.
I recently wrote a script to help me automate a very tedious task, and I thought it would be useful to break it down here. If you follow me on Twitter, you’re aware that I’ve been actively exploring and implementing IPFS to store NFT data for the Degenimals collection on OpenSea.
I will write about IPFS and NFTs at a later date, but it helps to serve as a backdrop for why I wrote the script.
About Shell Scripts
Linux shells are capable of very powerful scripting patterns, and they include a lot of common logic used in more complicated programming languages. Among these:
- Functions
- Loops
- Conditional statements
- Variables
- Inline string modification
- Strings, arrays, dictionaries
- Basic calculations
… and more!
I’m not going to get deep in the weeds with how to write shell scripts, but take my word that it’s quite powerful.
The OpenSea Script
The goal for the script was to download the avatar images associated with the minted Degenimals. I wanted to have a record copy of all avatar images, as well as their official NFT names, for easy publishing over IPFS.
The obvious and tedious solution is to visit the OpenSea collection and download them all by hand. But where’s the fun in that? Anything done by hand introduces the chance for human error.
Luckily for us, OpenSea has a documented API that we can use. The API is accessible over HTTP, which means that I can use CLI tools and shell scripting to handle it all for me.
Getting Started
The OpenSea API returns data in JSON format. It’s not super relevant to the shell scripting discussion, but knowing the output is critical so we can select the appropriate tool to process it. In our case, we use jq to process our received JSON data.
The first thing we need to do is retrieve some data from the API and see what we get.
First, let’s discuss how to retrieve HTTP data on the CLI. The tool you want is curl, which simply sends HTTP calls and sends the resulting HTTP data to your shell. You can test it by executing curl http://google.com (or similar) and seeing the resulting HTML.
My initial attempts at getting good data from OpenSea were frustrating, because the default call returns 20 assets at a time. I’m not skilled enough with jq to sift through a JSON array with multiple assets, so I edited the API call to request a single result. Here is that command:
devil@ubuntuVM:~$ curl -s "https://api.opensea.io/api/v1/assets?order_direction=desc&collection=degenimals&offset=0&limit=1"
{"assets":[{"id":30740264,"token_id":"47475974381565896501680608588151947420114181955589068365149607303160971395073","num_sales":0,"background_color":null,"image_url":"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw","image_preview_url":"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw=s250","image_thumbnail_url":"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw=s128","image_original_url":null,"animation_url":null,"animation_original_url":null,"name":"BowTiedHeifer","description":"The Official BowTiedHeifer\n\nMember of The @BowTiedJungle \n\nOwner of @BowTiedHeifer Twitter Account\n\nWelcome, anon","external_link":null,"asset_contract":{"address":"0x495f947276749ce646f68ac8c248420045cb7b5e","asset_contract_type":"semi-fungible","created_date":"2020-12-02T17:40:53.232025","name":"OpenSea Collection","nft_version":null,"opensea_version":"2.0.0","owner":102384,"schema_name":"ERC1155","symbol":"OPENSTORE","total_supply":null,"description":null,"external_link":null,"image_url":null,"default_to_fiat":false,"dev_buyer_fee_basis_points":0,"dev_seller_fee_basis_points":0,"only_proxied_transfers":false,"opensea_buyer_fee_basis_points":0,"opensea_seller_fee_basis_points":250,"buyer_fee_basis_points":0,"seller_fee_basis_points":250,"payout_address":null},"permalink":"https://opensea.io/assets/0x495f947276749ce646f68ac8c248420045cb7b5e/47475974381565896501680608588151947420114181955589068365149607303160971395073","collection":{"banner_image_url":"https://lh3.googleusercontent.com/2keCX7dNnAObsnKCi06-4djU4L8W8x_O_OqnRsXJbbP7yQR-lBaSxQTjwBlcPzThBvslU_yf5h0sqNzI55qS1MoKSuTU7IDJJcCS3Q=s2500","chat_url":null,"created_date":"2021-05-11T19:30:15.242986","default_to_fiat":false,"description":"The Official BowTiedJungle NFT Collection\n\nClaim & Mint your Official Jungle Avatar. Influenced & Inspired by BowTiedBull.\n\nWelcome, anon","dev_buyer_fee_basis_points":"0","dev_seller_fee_basis_points":"800","discord_url":"https://discord.gg/K6WfHphzBj","display_data":{"card_display_style":"padded"},"external_url":null,"featured":false,"featured_image_url":"https://lh3.googleusercontent.com/1Wj-MOUHtt3KiFXqmVtPY727PTEodWQWjFUCEeMQVYpujbqlsPSSV4E2K0AfnbGLR0gWbadQE5L0sU_maLp0MibI0X9qDW7z7QY2kPQ=s300","hidden":false,"safelist_request_status":"not_requested","image_url":"https://lh3.googleusercontent.com/kioJLaD1IOZadm-2zIc-WFZWFLWfdemun5ZTeb3tCAmSn6nBGTw3YEvS1pfoYHvJzHdxi87LQI5yRgAICMioFpMGkoc360p9s-mSfc0=s120","is_subject_to_whitelist":false,"large_image_url":"https://lh3.googleusercontent.com/1Wj-MOUHtt3KiFXqmVtPY727PTEodWQWjFUCEeMQVYpujbqlsPSSV4E2K0AfnbGLR0gWbadQE5L0sU_maLp0MibI0X9qDW7z7QY2kPQ=s300","medium_username":null,"name":"Degenimals","only_proxied_transfers":false,"opensea_buyer_fee_basis_points":"0","opensea_seller_fee_basis_points":"250","payout_address":"0x68f67301dc1d9a46211eda5c0408de7e44527423","require_email":false,"short_description":null,"slug":"degenimals","telegram_url":null,"twitter_username":"https://twitter.com/Degenimals?s=20","instagram_username":null,"wiki_url":null},"decimals":null,"token_metadata":null,"owner":{"user":{"username":"NullAddress"},"profile_img_url":"https://storage.googleapis.com/opensea-static/opensea-profile/1.png","address":"0x0000000000000000000000000000000000000000","config":"","discord_id":""},"sell_orders":null,"creator":{"user":{"username":"BowTiedRafiki"},"profile_img_url":"https://storage.googleapis.com/opensea-static/opensea-profile/33.png","address":"0x68f67301dc1d9a46211eda5c0408de7e44527423","config":"","discord_id":""},"traits":[{"trait_type":"Animal Group","value":"Mammal","display_type":null,"max_value":null,"trait_count":101,"order":null},{"trait_type":"Mammal","value":"Artiodactyla","display_type":null,"max_value":null,"trait_count":13,"order":null},{"trait_type":"Artiodactyla","value":"Cow","display_type":null,"max_value":null,"trait_count":2,"order":null},{"trait_type":"Degenimal ID","value":149,"display_type":"number","max_value":"150","trait_count":0,"order":null}],"last_sale":null,"top_bid":null,"listing_date":null,"is_presale":true,"transfer_fee_payment_token":null,"transfer_fee":null}]}
The key is the offset and limit parameters in the API call. By specifying a single item with offset and limiting the result to 1, I could retrieve the JSON data for a single Degenimal.
Filtering the Data
jq saves the day here by allowing us to filter for a particular item. In this case, the part I want is the image_url, which represents the avatar image stored on a Google server. You can look through it above and see that in this case, BowTiedHeifer’s image was stored at "https://lh3.googleusercontent.com/kioJLaD1IOZadm-2zIc-WFZWFLWfdemun5ZTeb3tCAmSn6nBGTw3YEvS1pfoYHvJzHdxi87LQI5yRgAICMioFpMGkoc360p9s-mSfc0=s120"
Rather than picking through this by hand, jq allows me to search for it by piping (remember that?) the output of curl into it and filtering for an element called image_url inside the assets array.
devil@ubuntuVM:~$ curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=0&limit=1"| jq '.assets[].image_url'
"https://lh3.googleusercontent.com/h7DgVuhJXR82CRKxGjmBQ6NcwpeAwFzYxgngGjGTbFMwDfjyx0inj7rBAio9lyiKqr0SyBBO-7wbi6pF8T4MwoncyZQjGNmy-JdUzw"
Nice, that worked! Now I have the image_url for the first Degenimal at offset=0 (in descending order).
OK, but the image_url has no indication that it belongs to BowTiedHeifer. I need some way to label the asset later, so let’s adjust our jq call to find the name instead of the image_url.
devil@ubuntuVM:~$ curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=0&limit=1"| jq '.assets[].name'
"BowTiedHeifer"
Storing the Asset
Excellent! Now we have a way to find the Degenimal name and get an URL for the image. Now we can get into using variables.
The format for a shell script variable is var=$(some command)
Let’s use name as our variable for the Degenimal, and then set it equal to the command we just discovered above.
devil@ubuntuVM:~$ name=$(curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=$i&limit=1"| jq '.assets[].name')
devil@ubuntuVM:~$ echo $name
"BowTiedHeifer"
Hmm… Looks good except that I don’t want those double quotes, so I edit the command by piping it into tr (remember that?) to delete all instances of ".
devil@ubuntuVM:~$ name=$(curl -s "https://api.opensea.io/api/v1/assets?order\_direction=desc&collection=degenimals&offset=$i&limit=1"| jq '.assets[].name' | tr -d '"')
devil@ubuntuVM:~$ echo $name
BowTiedHeifer
There we go!
Now I have some easily-reused commands to retrieve JSON data from OpenSea, filter for name, and filter for the image_url.
To retrieve something over HTTP, I use the wget command. It is called in the form wget [URL] to output to the shell, or wget [URL] -O [filename] to save to a file. Since I’m trying to download images, I use the -O option.
I already have image_url and name, so I simply send those variables into my wget command.
devil@ubuntuVM:~$ wget $image\_url -O assets/image/$name
Connecting to lh3.googleusercontent.com ([2607:f8b0:4005:807::2001]:443)
saving to 'BowTiedHeifer'
BowTiedHeifer 100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*| 50271 0:00:00 ETA
'BowTiedHeifer' saved
That’s a little noisy, so I check the wget man page to find the -q option (quiet output).
My command now looks like wget -q $image_url -O assets/image/$name.
Looping Through All Degenimals
Now that I have a nice set of commands to retrieve JSON data, filter for name and image_url, and save the file using wget, I’m ready to apply this to the whole set of Degenimals.
I can use the for i in ... pattern to construct a loop. Since the API returns assets starting at 0, and the maximum size of the Degenimal collection is 150, I build my loop: for i in $(seq 0 149). This will loop through all the Degenimals assets starting from the top and going until it’s reached 150 (I started from 0 so I only count to 149).
Then I re-write my data=$(curl -s "https://api.opensea.io/api/v1/assets?order_direction=desc&collection=degenimals&offset=0&limit=1" | jq ) command to use my new counting variable i instead of 0. In this way, the command will be re-written each time the for loop executes. It will start at 0, then go to 1, then go to 2, etc.
All Together
#!/bin/sh
mkdir -p assets/image
for i in $(seq 0 149)
do
# get JSON data from OpenSea API
data=$(curl -s "https://api.opensea.io/api/v1/assets?order_direction=desc&collection=degenimals&offset=$i&limit=1" | jq )
# get Degenimal name and image_url from retrieved data
name=$(echo $data | jq '.assets[].name' | tr -d '"')
image_url=$(echo $data | jq '.assets[].image_url' | tr -d '"')
echo -n "Processing Degenimal ($name)... "
# if assets already exist, skip
if test -f ./assets/image/$name
then
echo "asset exists, skipped"
else
echo -n "downloading... "
wget -q $image_url -O assets/image/$name && echo "done"
fi
# sleep to avoid hitting API rate limit
sleep 1
done
Running the script results in the following output:
devil@ubuntuVM:~$ sh ./opensea\_api\_process.sh
Processing Degenimal (BowTiedHeifer)... downloading... done
Processing Degenimal (BowTiedFarmer)... downloading... done
Processing Degenimal (BowTiedWestie)... downloading... done
Processing Degenimal (BowTiedSable)... downloading... done
Processing Degenimal (BowTiedSnipe)... downloading... done
Processing Degenimal (BowTiedNarwhal)... downloading... done
[output clipped]
Bang, done! A few minutes monkeying on the CLI and I’ve written a script that will save me hours of work.
Wrapping Up
Admittedly this post was a bit intense, so don’t worry if this went over your head. If you don’t care to play along with this one, please consider it a useful example of how many powerful things the Linux CLI allows you to do.
Could you do something like this on Windows? Mac? Your Phone?