Mass CTX Processing

A few weeks back, Ross Beyer presented my blog posts on autonomous HiRISE DEM processing at the HiRISE team meeting in Tucson, AZ. This brought about a question of could this be performed for CTX. Of course! ASP can be applied to do bulk processing for any of the missions it supports. Earth or any place ISIS and CSpice have defined coordinate system for. Just put in some safeties about run time into the processing scripts because ASP still occasionally goes mad and eats a whole bunch of processing time for no output. (Were working on it!)

Processing CTX stereo pairs however is in fact a little more difficult compared to the HiRISE processing I was doing before. HiRISE lists all their stereo pairs from their website or Dr. Shane Byrne’s website. There’s no equivalent for CTX. Luckily for me, some folks at UofA wrote PairendipityCTX (Chris Schaller?). They provided Ross and I a detailed report of overlapping files and other statistics. I cut out everything but the filenames for my own use and you can get a copy of the list with its 1,542 stereo pairs here.

Another difference was how these two missions stored their data on PDS. I can look at a HiRISE filename and work out its download path in PDS with no trouble. CTX on the other hand seems to have arbitrary volume breaks, thus the download URL is not predictable. My solution is a bad solution, but a quick solution. I wrote a python script that scraped PDS’s servers and identified all the CTX images it has and what their URLs are. I then just ‘grep’ for the URL in the processing scripts. Here’s the resulting text file that lists all of the 50,708 CTX images that existed at the time of my scraping. This is a mean trick because my script can make HTTP requests much faster than a human can. In a sense, I’m DOS’ing the PDS servers. So please copy my results rather than my methods.

Processing scripts

Previously with my autonomous HiRISE processing efforts, I just wrote a Bash script and then spawned it multiple times on my server using GNU parallel. That’s great, but looking around my office I saw a lot of unused computers that I’d like to do my bidding. What I wanted was a job management system like PBS, which is what they use on the super computer. But PDS is a little too heavy and sometimes cost money. So I instead found two alternative projects that seemed to fit the bill, Gearman and Celery. Gearman is the minimalist. Celery required a database backing and multiple ports open with each slave worker. I decided to use Gearman since it seemed simpler to learn.

My code for this little project is available on Github in my Mars3DGearman project. I hope to eventually add HiRISE support. Here’s how it plays out. All machines make a working directory that contains the CTX stereo list, the CTX url lookup list, and the folders DEM and DRG. One machine is designated the server, in my case, my workstation at home. It starts the only instance of ctx_processor.py with the help of a backing ‘gearmand’ executable (gearman daemon). All of the slaves then SSH back to my server and forward 4730, the port used by Gearman for communication. Then all the slave machines can start one or more instances of ctx_worker.py.

The part I haven’t worked out is how to send home the completed results to my main server. The ctx_worker script produces a DEM and orthophoto and then just dumps it locally into the DEM and DRG folder. Gearman does allow sending binary strings back the main server, but I’m betting a 100 MB long string would break something. I chose a different route. Since I already have all the slaves SSH’ing back to my main server, I decided to simply rsync all the slaves’ DEM and DRG folder back home. I don’t have to re-enter my password as I’ve enabled SSH ControlMaster which re-uses previous connections. For now, I just put that rsync in a watch command that fires every 2 hours. Alternatively it could be inside the ctx_worker script. A better bet would be to use SSH keys.

Another worthwhile improvement compared to my HiRISE processing scripts is the inclusion of a timeout method for each step. When it comes to CTX, if the correlation doesn’t finish in 2 hours, I’m not interested. This timeout is achieved through the run_cmd and process_timeout functions in the ctx_worker script. The Internet helped me a lot in figuring out how to make that a reality. Thanks Internet!

Results

These last few days I’ve roped 4 machines into doing my bidding. So far they’ve produced ~260 DEMs. 5 new DEMs completed just while I was writing this article. There are still some hiccups in the process. But when the stars align, I seem to produce over 50 new DEMs every day. I’m not going to show you all of them as that felt like a lot of work to post on to the blog. Instead I’m just going to show off a couple screenshots of some interesting places in Google Mars. The color ramp is a little funky because someday I need to learn to reference everything against the Mars Aeroid and not just the sphere.

Not everything looks as great as those screenshots. Here are some places that failed to correlate. I’m not sure what went wrong and I unfortunately don’t have the time to investigate.

In conclusion, this is just another proof of concept of what is possible. I hope that someday someone else will attempt this and do a better job than me. ASP is not perfect, but it can achieve a lot of processing on its own that could be beneficial to the scientific community.

Related

Shean, D. E., et al. “MRO CTX Stereo Image Processing and Preliminary DEM Quality Assessment.“ Lunar and Planetary Institute Science Conference Abstracts. Vol. 42. 2011.

David Shean, MSSS, and Larry Edwards actually already attempted this once before! In the above abstract you can see that they went above and beyond my little demo and processed 1180 stereo pairs. They also developed some initial steps for registering the output DEMs to MOLA and plotted the relationship convergence angle has on the outcome of a stereo pair.

MER Data Introduction

Getting data from the NASA Planetary Data Services (PDS) can be a little intimidating. The most import thing a person needs to understand about the MER data is that it is sectioned into individual sites that the rovers visited during their trip. These sites are localized on interesting features like an individual bay into a large crater or a particularly large bolder. These sites are further subdivided into sols (Martian days) at the site. When we search for images we limit them by both sites and sols.

Gathering Data

It is now time for us to select a site for which we want to render in 3D. I find MER’s Analyst’s Notebook as a good tool for this task. Here’s what I did:

  1. Go to: http://an.rsl.wustl.edu/mer/
  2. Click on “Opportunity”.
  3. To find a site: click on the “Map” icon on the toolbar, and then click on “Traverse Map”.

Each one of the black dots represents a site location. The number after the slash is a sol number. I compared this map to Google Earth (GE) and decided that I wanted to render “Cape Agulhas”. You can do that too in GE if you click on the flag icon for a MER and select “load rover way points”. Just note that the map on GE doesn’t show as much information as the traverse map on Analyst’s Notebook.

  1. On the “Traverse Map”, click “91/1673”.
  2. Select only Navcam data products on the left.
  3. On the Data Products drop down box, select “Show for sol 1674”
  4. Click “Redraw Map”.

From here we can see that the Navcam did a full panorama. We are now ready to start downloading data from PDS. You can download from Analyst’s Notebook, however I was only able to download a single image at a time. On NASA PDS we can download a ‘wget’ script that will download all Navcam images for us in one go.

  1. Go to NASA PDS’s image search for MER, http://pds-imaging.jpl.nasa.gov/search/search.html#QuickSearch
  2. On the left, under “Select Instrument(s):”, select “NAVCAM.
  3. Next to “Instrument Host ID”, select “MERB/MER1/Opportunity”.
  4. Next to “Product Type”, select “EDR”.
  5. Next to “Image Type”, select “Regular”.
  6. Next to “Eye”, select “LEFT”.
  7. Next to “Planet Day Number”, type 1674 for both the Min and Max text boxes.
  8. On the left, click “Get Results”.

You now have a listing of all the Left EDR NAVCAM. We want all of these; so on the left select ‘WGET’ under download products. Then click download. Move your downloaded ‘atlas_wget_script’ to your work directory. Here is how you start the download in the terminal:

cd $YOUR_WORK_DIR
source atlas_wget_script

Later on I can show how to produce 3D maps from this imagery. However this seems like a good start.

Fly by night Cassini ISS processing

Anytime I want to process Cassini ISS data, I always run into problems. This is largely because the documentation is lacking. Today is the day that is going to change. Here are my notes on how to extract ISS data of Iapetus, all for the low cost of free. I assume you have USGS’s ISIS3 installed.

Iapetus, file N1483152391_1

Downloading from NASA PDS

As usual, all NASA data can be pulled free from their PDS (Planetary Data System). Since I’m looking for Iapetus, I click ‘Saturn’ and then ‘Cassini Image Search’. This will bring you to the Cassini Image Quick Search page. On the left side of the page, select the ISS instrument. Then select EDR under product type. EDR products are raw images while RDR are processed products like maps. Finally under target name, select the option Iapetus. Click ‘Get Results’ on the far right. At the time of writing, there was 4025 records, or images available.

Here’s where things get a little messy. A lot of these images are of Iapetus from very far away. So far away that they wouldn’t be good for my projects. I’d like to sort these images by their distance. On the far right, under SELECT PARAMETERS FOR REPORT OR TABLE COLUMNS; I can add the column ‘Target Distance’. On the new column, I can click the up arrow and hopefully it will sort the images by who is closest. However it doesn’t work. In the middle of the page; PDS says “508 of 4025″ products. The page is only sorting 1/8th of the total images of Iapetus. To correct this, keep clicking the ‘Get More’ button until it says “4025 of 4025″ products.

At this point we are ready to start downloading products. On the far right of the page, download the CSV file of the records they found along with the WGET product. When you open up the CSV file you’ll notice that they didn’t keep the images ordered in distance from target. Lame, I know. However, we use Linux and we can pull ourselves out of this. Here’s the bash commands I used to sort the file. Gurus can probably do a better job of this than me.

cat atlas_report.csv | awk -F "," '{print $3 $4}' | sort -n -k 2 > sorted_files

If you open up sorted_files in your favorite text editor, you can clean up the stray lines that were mis-sorted. I chose to keep the files with distances marked ‘-1.0e3′. I also hand deleted the lines where the distances were marked greater than 300 km out from the moon. This left me with 758 names of images that are pretty close to Iapetus. Now we need to pull these 758 names from the atlas_wget_script so that I only download the images I want. However when you look in that script, you’ll notice that they name the images slightly different from what we saw in the atlas_report.csv. Here’s an example:

1_N1466586085.118 versus N1466586085_1.IMG

Here’s how I filtered sorted_files down to only that mantissa like bit I wanted and then pulled out the respected lines I wanted out of atlas_wget_script.

cat sorted_files | awk '{print substr($1,3,11)}' > wanted_keywords
cat atlas_wget_script | grep -f wanted_keywords > wanted_get_script

Hooray! Finally we are ready to really download imagery files! If you look inside wanted_get_script, you’ll see the download commands for all the images of Iapetus that I want. To start downloading:

source wanted_get_script

Processing with ISIS 3

You probably think it’s all kisses and hugs from here. You have a direct full of IMG and LBL_label files do ya? Well the command we want to use is ciss2isis. Well then, don’t let me hold you back, read their instructions and give it shot on one of the images you downloaded.

ciss2isis from=N1483150900_1.LBL_label to=test.cub

What it broke?

**I/O ERROR** Unable to read PVL file [/Volumes/Data/Saturn/CASSINI_data/Iapetus/temp/N1483150900_1.LBL_label]
**PVL ERROR** Error in pvl on line [166]
**PVL ERROR** Unable to read keyword [DESCRIPTION    = "For the packet from which this telemetry header was Extracted:  the first line of this packet is a continuation of a line begun in the previous packet. OBJECT = COLUMN NAME = NULL_PADDING DATA_TYPE = MSB_UNSIGNED_INTEGER START_BYTE = 61 BYTES = 475 END_OBJECT = COLUMN END_OBJECT = TELEMETRY_TABLE OBJECT = LINE_PREFIX_TABLE INTERCHANGE_FORMAT = BINARY ROWS = 256 COLUMNS = 7 ROW_BYTES = 24 ROW_SUFFIX_BYTES = 512 OBJECT             = COLUMN NAME               = LINE_NUMBER DATA_TYPE          = LSB_UNSIGNED_INTEGER START_BYTE         = 1 BYTES              = 2 DESCRIPTION        = "The image line number of this record.  Maintained at]
**PVL ERROR** Keyword has extraneous data [The image line number of this record.  Maintained at] at the end

This looks darn scary. However if you download the original LBL files from the PDS manually without using the atlas_wget_scripts ‘s version, the problem becomes clear. The download script was accessing a new label file that had a poorly inserted ‘EXTENDED_ISS_SCIENCE_HEADER’. This was probably done by some batch processing job at PDS or Cassini imaging lab. This mistake on PDS managed to delete some of the lines defining the end of objects. This caused the ISIS3 command ciss2isis to freak out an die.

Yet don’t worry .. this can be corrected. I have written a patch for these LBL_label files that will make then usable. In a new file, write the following contents.

--- N1483150900_1.LBL_label     2010-10-19 19:46:20.000000000 -0700
+++ N1483150900_1.LBL_corr      2010-10-19 19:45:01.000000000 -0700
@@ -99,6 +99,7 @@
BIT_DATA_TYPE  = BINARY
START_BYTE     = 1
BYTES          = 60
+END_OBJECT     = COLUMN

 OBJECT          = BIT_COLUMN
NAME           = CAMERA
@@ -144,7 +145,8 @@
BITS           = 1
DESCRIPTION    = "For the packet from which this telemetry header was
Extracted:  the first line of this packet is a
-                    continuation of a line begun in the previous packet.
+                    continuation of a line begun in the previous packet."
+ END_OBJECT   = BIT_COLUMN
OBJECT = COLUMN
NAME = NULL_PADDING
DATA_TYPE = MSB_UNSIGNED_INTEGER
@@ -209,6 +211,8 @@
DATA_TYPE          = LSB_UNSIGNED_INTEGER
START_BYTE         = 9
BYTES              = 2
+END_OBJECT         = COLUMN
+
END_OBJECT = LINE_PREFIX_TABLE
OBJECT = IMAGE
      LINES = 256

I saved the above as correction.patch. It can then be applied to all LBL_label files to make then work.

echo *label | xargs -n1 echo | xargs -n1 -I{} patch {} correction.patch

You will now find that ciss2isis works. Here’s how I do that with multiprocessors.

echo *label | xargs -n1 echo | awk -F "." '{print $1}' | xargs -n1 -I{} -P8 ciss2isis from={}.LBL_label to={}.cub

You now have a directory with cub files. You can view those with the qview command or export them with isis2std. There’s still more that needs to be done inorder clean up this imagery. I won’t tell you today, but I can hint what the next few steps will be. Have fun!

Iapetus plus stars