Harvesting ArcGIS Online Data and Maps Metadata

This short article describes a process where Python was used to harvest metadata from a list of identified ArcGIS Online maps and the maps’ data services. The data were logged to MySQL (with pymysql); a PHP web search and discovery page was created.  The process allows for keyword searching in titles and descriptions of maps and data layers.

Why harvest metadata?

This approach was used as our collection of 150 maps is housed in several AGO organizations with data services spread across even more.  These data and maps are designed for student use and vetted, making school curriculum authors interested in search and discovery of “good” data.  Essentially, we have a target population that is keenly interested in a subset of scattered data and maps.

Who might use this approach?

This article may be of interest to developers needing to create a search solution across a specific list of maps and constituent data services.

The Approach

Using the ArcGIS Python API 1.5.2 and a prebuilt list of mapIds, a script was built that iterates over the list, logging titles and descriptions for the maps.

with open('data/maps.csv') as f:
reader = csv.reader(f)
for map in reader:
result = gis.content.get(map[0])
web_map_obj = WebMap(result)

Using the layers attribute now available for the map object, we then loop over all the layers in the map (above), also logging titles and descriptions.

We then log the data to a MySQL table (with a custom function), like:

  url='http://www.maps.arcgis.com/home/webmap/viewer.html?webmap=' + result.itemid,

Then loop over each layer and log to the table, slightly changing parameters as necessary.

You can access the script at GitHub below. The Python code, a sample CSV input file, and a sample SQL script for generating the MySQL table are included.

github  https://github.com/trbaker/mapHarvest

Python and Cron to Refine SSO Account Creation

Single sign-on offers great benefits for schools using ArcGIS Online (AGO) and other SAAS products. Today, one limit of SSO is that the SAML implementation offers an efficient but subset of features one might need in order to effectively manage a large user base.

For example, I was approached by a large K12 educational entity (greater than 10k) that wanted to implement SSO for user account creation and authentication.  However, they also wanted teachers to have publisher roles and students to have user roles.  The current enterprise login system in ArcGIS Online casts everyone into one role. Enter Python!

Begin by setting up your workstation or server, ideally one with the capacity to schedule tasks (Cron, Windows Task Scheduler, OSX launchd or iCal). When the script is ready, you will need to use a scheduler to run the task repeatedly. This is in part because there is no way to trigger the script on an SSO event in AGO (at least today).

The script is built on a Python stack including: ArcGIS Online API for Python 1.5.1, Python 3.6, and Anaconda 4.4.  You can use Jupyter to create the script but is unnecessary for regular or scheduled execution. See API and stack installation details.

The script retrieves all the user accounts in the target organization and then filters out the unnecessary users. In this script, three filters are applied:

1.    Ignore accounts without the autogenerated appended organization name (case-sensitive). Check for the appended org subdomain (applied automatically by SSO and other auto-generated username mechanisms). In this case, the pattern we want looks like: tbaker_SchoolDistrictX, where “SchoolDistrictX” is the organization’s subdomain.  This doesn’t guarantee the account was created via SSO but can rule out manually created accounts, depending on username policy in the organization.

2.    Ignore user accounts more than one week old.  This also prevents manual account edits (to accounts over one week old) from being reverted accidentally by the script.

3.    Use a regular expressionto sort the usernames into teacher versus student.  In this example, student usernames all contained six consecutive digits where teacher accounts did not.  Student accounts were filtered out since all new SSO accounts were created at the lower student level (user).  When a username without six consecutive digits was found, it was passed through.

Get the script at GitHub. >>

Once a user passes all the filters, the “update_role” function is used to upgrade to built-in role, “org_publisher”. This script would need modification if using with a custom role.

Any of the filters can be changed or removed based on organizational need. The regular expression in filter three will most certainly have to be modified by everyone.  Many will want to run the script daily or even hourly, requiring a commensurate time change in filter two.

Of course, be sure to comment out the “update_role” function until you have completely tested the script in your environment. I generally inset a print statement in place for feedback during testing.   Like most scripts, there’s no “Undo” button.

For initial deployment, I saved the script to a “.py” file and scheduled it to run from a laptop on the corner of my desk however very shortly I’ll deploy to a dedicated Amazon Lightsail Ubuntu server.  Enjoy!


Read more:

Five Reasons to Use Single Sign-On in Schools

Single sign-on uses a school district’s identity provider software (such as Microsoft Active Directory or Google G Suite) to support student authentication on devices, when accessing the school network, or when using district software.  Single sign-on provides three broad categories of benefits to teachers: increased instructional time, stability and support across the school, and reduced teacher liability.

When Single Sign-On (Enterprise Logins) are turned on by the district technology staff, students and teachers need only to click-through the regular district sign-in dashboard (such as the G Suite drop-down dashboard).  The first time a student or teacher clicks on the icon for ArcGIS Online, their account is created and they are automatically logged in using their school district credentials.  When teachers who previously used bulk/batch methods of creating student accounts see how fluid and perfect this approach is, we hear words like “incredible” and “awesome”.

Aside from the “wow-factor”, we’ve noticed at least five reasons most districts should consider moving to single sign-on with ArcGIS Online.

Focus on Instruction

1.    Teachers have minimal (if any) user account management with SSO.  Without SSO, how much time have you spent creating (and managing) student accounts? How long does it take to reset the passwords for students who have forgotten their login?

2.    One-click and students are in ArcGIS Online – requiring zero-time during a class period, allowing more time for instruction to go further.

Stability and Support

3.    Single sign-on moves GIS from the periphery in a classroom to mainstream – as everyone in the school/district can automatically access AGO.  Don’t worry, you can request more users or credits for your school organization at no cost from Esri (assuming responsible prior usage).

4.    Reduce dependence of ArcGIS Online on long-term presence of one particular teacher in a school. This means that spatial habits of mind and mapping tend to become institutionalized – allowing you to better establish a program that will stand the test of time.


5.    Teachers don’t have to assume the liabilities of making IT security decisions or ensuring student privacy. When the technology department implements SSO, they’ll conform logins to district policy – protecting everyone! Do your students’ ArcGIS Online accounts have passwords that conform to district password policy?

While some teachers report working with district technology staff to be challenging (and vice versa), the benefits of SSO vastly outweigh the costs in most cases.  If you’re interested in moving forward, talk to your district technology or edtech staff.  Technology directors, talk to your social studies or science curriculum directors and make a plan today.

For more information on the state of single sign-on with ArcGIS Online in K12 education, visit the K-12 landing page at http://esriurl.com/K12SSO.

Hello world!

This is your very first post. Click the Edit link to modify or delete it, or start a new post. If you like, use this post to tell readers why you started this blog and what you plan to do with it.

Happy blogging!

Categories Uncategorized