Python and GeoProcessing on Linux, part 1

After reading this ESRI blog post, I decided to give this a try with ArcGIS 10.  I love playing with Linux, and I’m pretty good with Python, so this seemed like a fun experiment.

I set up a VirtualBox VM with Fedora 13 (32-bit).  Next, I installed Eclipse.  The Fedora version of Eclipse installs PyDev for you, so a separate install of that is not required.  Should you want to update PyDev, inside Eclipse go to Help->Install New Software.  Click the Add button to add a site.  The location is:  http://pydev.org/updates.  Eclipse includes PyDev 1.5.5, and the updates list 1.6.0 as being available.  I have not tried the upgrade yet, so do this at your own risk!

Next comes the ArcGIS Engine installation.  Pretty straight-forward.  Mount the CDROM (or ISO, or however you want).  You must install the EngineRT first, then the ArcObjectsSDKJava.  If you try to install the ArcObjectsSDKJava first, it will give you an error message about the Runtime, and bail. 

To install the Engine Runtime:

$>cd /media/ESRI/linux/EngineRT

$> ./Setup

 

Walk thru the wizard, which is very simple.  I installed the Single User setup.  I got a warning about my system being “not supported”, but we knew that already. 

To install the ArcObjectsSDKJava:

$>cd /media/ESRI/linux/ArcObjectsSDKJava

$> ./Setup

At the end of the ArcObjectsSDKJava, you will be asked if you want to Register your software.

Select the checkbox, and select Done.  You will then walk thru the Authorization wizard, which appears to be running on Wine (it has an XP interface!) 

Make sure you authorize both Engine Runtime and Engine Developer Kit.  You should get your successful confirmation at the end of the process.  You can confirm your license with the Administrator window.

To finish part 1 of this post, we are going to use a terminal to verify that everything is installed correctly.  Start a terminal by going to Applications->System Tools->Terminal.

In your terminal, we will source the “init_engine.sh” (assuming you use Bash).

$>source ~/arcgis/engine10.0/init_engine.sh

$>

You should not see any error messages.  Now, start a python session:

$> python
Python 2.6.5 (r265:79063, Apr 27 2010, 11:08:55)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import arcpy
>>> print arcpy.ProductInfo()
EngineGeoDB
>>>

Success!  We can also list the tools available:

>>> import arcpy
>>> tools = arcpy.ListTools("*_management")
>>> for tool in tools:
...     print tool
...

You should see a list of tools in the _MANAGEMENT toolbox.

There is some definite quirkiness going on, and I haven’t had time to research much of it.  But if you try this:

>>> print arcpy.Usage(tool)

unicode(string [, encoding[, errors]]) -> object

Create a new Unicode object from the given encoded string.
encoding defaults to the current default string encoding.
errors can be 'strict', 'replace' or 'ignore' and defaults to 'strict'.

However, you can do this:

>>> print arcpy.Usage(tool.encode('utf-8'))

SynchronizeMosaicDataset_management(in_mosaic_dataset, {where_clause}, {NO_NEW_ITEMS | UPDATE_WITH_NEW_ITEMS}, {SYNC_STALE | SYNC_ALL}, {UPDATE_CELL_SIZES | NO_CELL_SIZES}, {UPDATE_BOUNDARY | NO_BOUNDARY}, {NO_OVERVIEWS | UPDATE_OVERVIEWS}, {NO_PYRAMIDS | BUILD_PYRAMIDS}, {NO_STATISTICS | CALCULATE_STATISTICS}, {NO_THUMBNAILS | BUILD_THUMBNAILS})
Synchronize selected rasters in the mosaic dataset

If you get some error messages, leave me a comment and I’ll try to answer (I got a ton of errors before I had success).

In Part 2 (soon to follow), I will document how to configure Eclipse and PyDev to work with ArcPy.

Continuing the Battle with Apple

In my last blog post, I went into some details about how my iTunes account had been hacked, and some $350 charged to my credit card. I figured I would write an update as to what has happened lately.

I contacted my credit card company, and had to close the account and get new credit cards issued. That’s always a pain because I have items like Netflix that get billed to that account monthly.

Once I got my credit card, and regained access to my iTunes account, I was met with a message that I had an outstanding balance of $16.22. This was an unauthorized charge that had to be taken care of before my account would be re-enabled (meaning, I can’t even download FREE apps until this is resolved). So I emailed back to Apple Tech Support, and received this response:

I would then ask that you pay for the currently pending order. While I understand that you did not make the purchase, it will need to be paid before you can make purchases on your account again. However, once paid, you would then ask that you reply to this email to let us know so that we can issue you a one time exception refund for this order.

This puzzled me, as I was previously told that Apple does not have the ability to generate refunds. Now I am being told that they DO have that ability, and that I MUST PAY FOR AN UNAUTHORIZED CHARGE with my brand new credit card. Ugh. Really?! Just REMOVE THE UNAUTHORIZED CHARGE from my account…

Again, Tech Support is oh so helpful and understanding.

I know you may be hesitant to add your credit card back to your account, but I assure you that once you do, and the previous order has been paid for, I will be able to simply refund you for it as a one time exception. In the end, you will not lose any money from the order and will regain your account, and the ability to make use of it again.

Lastly, Tech Support gave me NO OTHER OPTION. If I ever want access to my iPhone apps again, I have to pay for this unauthorized charge, and then rely on them to refund it and re-enable my account.

Well, sadly and reluctantly I paid for my $16.22 of unauthorized pending charges on 5/2/2010. And since then, I have heard NOTHING back from Apple Tech Support. I still do not have access to my account. I cannot download new apps, even FREE ones. I cannot download upgrades to my existing apps.

I urge everyone to consider buy their music and devices from somewhere other than Apple. Their lack of security and customer support is simply not worth the hassle.

UPDATE: As of 5/14/2010, this issue has finally been resolved with Apple. It took a complaint via the Better Business Bureau, but ultimately I have received my refund and my iTunes account has been re-enabled. I am still displeased with Apple's lack of security regarding iTunes accounts. When I suggested some ways for Apple to make their iTunes accounts more secure (i.e., verifying a new email address, emailing if/when security options change, etc), they simply replied with:

I encourage you to use the iTunes Feedback page to submit your comments:
http://www.apple.com/feedback/itunesapp.html

So I encourage everyone to provide similar feedback (or stop using iTunes altogether).

Apple’s iTunes store lacks fraud prevention

Over the past few years, I have embraced some of Apple’s new technology. Specifically, I bought an iPhone 3G and have loved it.  For better or worse, it has given me a new sense of connectivity to the world.  I can use VPN connectivity to monitor my internal or customer databases from anywhere with a 3G connection.  And the games sure keep the kids entertained while eating at restaurants where the service is slower than anticipated.

The positive side of the iPhone is undeniable.  However, I have now found that the negative side of Apple is also undeniable.  Let me explain.

First, I find it appalling that you cannot make a simple phone call on an iPhone until you register your device thru iTunes.  Since I have always been a fan of Napster, I did not have an iTunes account.  So I was forced to open an iTunes account and give Apple my credit card information BEFORE I CAN EVEN MAKE A SINGLE PHONE CALL.  What?!  Really?? Horrible business decision.

Fast forward a year, and I have upgraded to an iPhone 3GS, handing down my iPhone 3G to my wife.  (She originally got a brand new 3GS, but had it stolen within 3 weeks.  Apple and AT&T were incredibly reluctant to help recover the stolen device, but that is another blog post.)

One day, my App Store icon on the iPhone 3GS tells me that I have 5 application updates available.  So I proceed to attempt to download them by entering my iTunes password.  Upon entering my password, iTunes tells me that the Apple ID or password is invalid.  Several failed attempts later, iTunes takes me to https://iForgot.apple.com, to help me recover my Apple ID password.  When I enter my Apple ID (the same Apple ID that was used by the App Store app to inform me of the updates in the first place), the website tells me:

“The Apple ID you entered DOES NOT EXIST.”

Excuse me?  My iPhone is using that ID.  iTunes on my Windows 7 desktop is using that ID.  How does it not exist?  So I promptly email Apple technical support.  Their reponse:

“After careful review, I found that you are being told “<your email address>” does not exist because, quite simply, it doesn't. I noticed that your account, which was formerly named “<your email address>” was actually renamed to be "<non-existent email address>".

Um, WHAT?  Why would I rename my account?  How and when did this happen?  Again, Apple’s tech support response:

“Regrettably however, I am not able to provide any information about account activity, including when this change was made. However, I am more than happy to get this all sorted out for you.”

So after answering a myriad of security questions, Apple Technical Support was kind enough to reset my password on the account using the <non-existent email address>.  When I logged into my iTunes account, I was met by a prompt to download my pre-order of Avatar.  Again, WHAT?  I’ve not pre-ordered Avatar.  So I look at my Order History, and my jaw hits the floor.

If my calculator is correct, between 4/15 and 4/16 I purchased $353.51 worth of new content from iTunes!?!

Holy Shit!?!  Really?  My single largest purchase from iTunes in the previous 2 years was $5.98.  Suddenly, I went on a spending spree of $353.51?! 

Oh, and that happens to coincide with the following account activity:

• Changing the account email address / name;
• Changing the account password;
• Changing the account security question;
• Changing my birthday on the account (which Apple uses as a secondary security question)

How much more of a fraud alert does someone need?  Apple did not send me a single confirmation email when any of my security information was changed.  They did not confirm the new email address (as the new one obviously doesn’t exist).  They did not detect a nearly 60 X increase in my spending habit.  And they didn’t bother to realize THAT ALL HAPPENED AT THE SAME TIME.

When I alerted Apple to what happened, here is their response:

“I now understand you are concerned about purchases that were made with your iTunes Store account without your permission or knowledge. I truly am sorry to hear that this has happened to you.

To prevent further purchasing, I have disabled your account. Please note that your iTunes account can be enabled in the future by providing specific information to iTunes Store support.

I urge you to contact your credit, debit, or payment card issuer as soon as possible to inquire about canceling the card or account and removing the unauthorized transactions. You should also ask them to launch an investigation into the security of your account. The iTunes Store cannot reverse the charges.”

iTunes CANNOT REVERSE THE CHARGES?  I am informing Apple of fraudulent charges, and their response is “we disable your account and ask you to cancel the credit card, otherwise we are keeping the money”.  I find this just slightly unacceptable.

Even better is this gem at the bottom of their email:

“I hope the information I've provided proves to be helpful to you Jeff, and that that you are able to resolve this matter with the help of your card issuer. When you are confident the matter is resolved, please return to the Apple Account Information page, click the Edit Payment Information button, and enter your billing information.”

Why YES INDEED, the information you have provided has been more than helpful. You want me to log into my compromised account, and GIVE YOU MY NEW CREDIT CARD INFORMATION!?   Hey Apple, GO FUCK YOURSELF.

Running Cygwin SSH as a Service

This is not a complete installation guide, there are plenty of those available via Google.  These are my hints/tips, so I remember how to do it next time.

1)  Assuming you already installed Cygwin, along with SSH.  You need to run the config script:   /usr/bin/ssh-host-config

2) This will call CYGRUNSRV.EXE, which creates the service for you. 

3) I don’t want to run it as a local (CYGSRV) user, I want to run it as one of my existing users.  To change that, go into the service and change the logon parameters.  Stop the service.  Then run these three commands from a bash shell:

chown <user> /var/log/sshd.log

chown –R <user> /var/empty

chown <user> /etc/ssh*

Now restart the service, and it should be running as <user>.

Creating ISO files on Windows

WARNING:  This is a NON-GIS-related post.

In the past 6 months, I’ve started using a lot more “digital media”.  We bought a Canon Vixia HF200 HD camcorder for recording the kids.  We bought a Canon Rebel XTi digital SLR for, well, taking pictures of the kids…

With all this digital media, one thing I’ve started doing is capturing “raw ISO files” of my media.  So if I record an event on the Vixia, I will take the SDHC card out, and immediately create an ISO image of the whole card.  That way, should anything happen, I can always go back directly to the source.

I tested a lot of tools for dealing with ISO files on Windows.  FREE ISO Creator by minidvdsoft worked very well, until it started nagging me about buying it.  LCISOCreator also had some positives.  But in the end, I found that ISO Recorder V 3.1 worked the best for my workflow.

When I insert an SDHC card, I literally right click on the drive letter (H: in my case), and select “Create ISO Image File”.  I give it an output ISO path, and I’m done.  Couldn’t be any easier.

As a side note, I use SlySoft’s free Virtual Clone Drive to mount my ISO files, import my AVCHD files, and create my videos.

Both of these free pieces of software support Windows 7 x64. They should be a part of any geek’s toolkit.

Find recent files via python

Here is a quick little python blurb to find any files in a given directory within a certain time.  I use this in my nightly scripts to find my lastest database backups, exports, shapefiles, etc.   Hope it helps.

 

import os, datetime, stat

NOW = datetime.datetime.now()
TWELVE = datetime.timedelta(hours=12)
FORMAT = "%a %b %d %Y %H:%M:%S"
DIR = "D:/My/Full/Path/"

for x in os.listdir(DIR):
    int_time = os.stat(DIR + x)[stat.ST_CTIME]
    if datetime.datetime.fromtimestamp(int_time) > (NOW - TWELVE):
            # Do something with x here (ie, copy it somewhere)
            print x + " --> " + datetime.datetime.fromtimestamp(int_time).strftime(FORMAT)

ESRI Developer Summit 2009

I am finishing up my trip to Palm Springs for the ESRI Developer Summit.  Fantastic week, learned a LOT, met several new technical contacts and had an enjoyable time.  These are my Dev Summit highlights.

• The biggest news I got out of the summit was that the GDB schema in ArcSDE 9.4 is going from 35+ tables down to 4.  They are going to essentially use an Open XML schema to describe all the GDB objects (object classes, feature classes, feature datasets, relationships, domains, etc.).  This is going to break a lot of existing sites that read those GDB tables (yes, even those ESRI tells us not to do it, we still do). 
• I met with the GDB team Wednesday morning, and they confirmed that they are going to use XML field types, rather than BLOB.  This alleviates my biggest fear.  Next fear becomes scalability.  ESRI already requires a lot of exclusive locks for doing most anything in ArcCatalog.  Is only having 4 tables going to place additional locks on those tables and require MORE exclusive locks?  Let’s hope not.
• ArcGIS Explorer 900 is going to be fantastic.  If you haven’t seen a demo of it yet, you should.  I can’t help but wonder how many ArcView licenses this will steal away.  It’s a free product with a LOT of functionality.  Beta release coming soon.
• Lots of emphasis on python this year.  At 9.4, ESRI is replacing the “command line” in ArcGIS with a full python interpreter.  Nifty.
• Had lunch on Tuesday with a group of GDB people, the main topic was Managing the GDB.  Specifically, how to deal with data modeling and multiple geodatabases.  ESRI has assigned a resource to deal with this problem, if you want his contact info so you can have some input, then leave a comment for me with your email, and I will forward you his email.
• One issue I’ve dealt with in the past is that if you use an Oracle Export against ST_GEOMETRY, the resulting IMPORT has issues with indexes.  ESRI KB Article 34342 deals with this.
• Spatially clustering data storage of ST_GEOMETRY tables can provide significant performance improvements.  ESRI KB Article 32423 gives a procedure for clustering your data.

These are my initial, still-in-Palm-Springs highlights.  Will clean them up a bit when I get back to Charlotte.