A Raspberry Pi IoT to collect and post field data

This is the third blog post in a series that uses the Raspberry Pi as an Internet of Things device to collect environmental data (with a DHT22 weather sensor) and post the data in real-time to an ArcGIS Online feature service.  The post below is a highly synopsized version of a chapter in the forthcoming third edition of the book, Tech-enabled Field Studies.

Begin by setting up a Raspberry Pi with the DHT22 sensor..  See The Pi Becomes a Data Collector

Use the Pi’s command prompt to make the following installations and configurations:

  1. pip3 install virtualenv
  2. virtualenv datalogger
  3. cd datalogger
  4. source bin/activate datalogger
  5. pip3 install Adafruit_DHT
  6. pip3 install pandas
  7. pip3 install arcgis – -no-deps

Note: Step 6 above (Pandas) may take several minutes. The above steps create a virtual environment in Python called datalogger and install three libraries into that environment including a streamlined ArcGIS API for Python.  When you call the weather script below, you will need to reference it from this environment, described below.

Continue the set-up by:

  • Configuring the ArcGIS Online service as described in the previous blog post, A Time-aware Geo Data Bucket.
  • Download the sample Python file that reads data from the DHT22 and posts to ArcGIS Online. The Python file is here or download/clone the entire project here.  be sure to update the python file with your information, as described in the file.
  • Put the Python file in your datalogger directory to help you remember that it must be run from that environment.  Once you have added your data to the python script and have the ArcGIS Online service running, you can run the script by using the command:
    • /home/pi/datalogger/bin/python3 addwx.py

The service does have a date/time attribute and the service can become time-aware.  Once the service is added to a map, you may also choose a symbology that you prefer.

Finally, on the Pi, you may choose to set-up a Cron job to schedule the script execution on a reoccurring basis.  Do a quick Google search to find tutorials on how best to do this.  Enjoy!

 

 

 

The Pi Becomes a Data Collector

The Raspberry Pi can connect directly to sensors.  From temperature and humidity to light and motion to sound and well beyond, these add-ons can be sold individually or in large kits.  This ability to read and process data from a sensor turns your Raspberry Pi into an instant IoT device – for under $50.

Get a Sensor

In this project, a 3-pin DHT22 sensor with capacitive humidity sensor and a thermistordht22 (for temperature  and humidity) will be used.  If ordering from Amazon you can either get sensors cheap or fast – bur rarely both.  I purchased the $9.99 Gowoops version (2 sensors at $5 a sensor) of the DHT22 and received it in about two days.  I also purchased the $3.68 Sodial version (1 sensor) and received it in about one month. Make sure the sensor you order comes with cables/wires to connect the sensor to the Pi.   If you’re lucky enough to have a MicroCenter or similar store in your city, you might also look there.  There doesn’t appear to be much difference in sensor quality (in my two sensors).  What I paid for was speed of delivery.

Pi Compatibility

The easiest way to connect a sensor to a Raspberry Pi is by using the built-in header, available on current Pi models (except the Pi Zero).  On the Pi 3 B+, the basic GPIOPi3BPlus header is a 2×20 male pin connector panel that is built-in to the Pi’s board (visible on the top edge of the board pictured).  Each pin has an address; some pins are designated for data input or output, others for power, and some are ground.  Connecting the sensor is a matter of getting the three DH22 wires connected to the correct pins.

Generally for this type of project, I would recommend the Pi 3 B+, due to its processor speed and built-in hardware.  A Pi Zero for example does not have a built-in header and one must be soldered in place first.

Connect the Sensor

  1. Turn the Pi off and unplug the power.
  2. Remove the case. Position the Pi’s board so the header sits at the top edge (away from you).  Look at the GPIO header diagram below. Locate pin 1, which is on the left side of your board in the row closest to you.GPIOB-Plus
  3. Connect the 3 colored jumper cables from the sensor to the GPIO.  Cable colors will vary.  Technically, on the header, it doesn’t matter which power, ground, or GPIO pins you use, as long as you know the GPIO number of the data cable.  It will need to be used in your test program.  I recommend:
    1. Power cable: sensor (+) to header pin 1
    2. Data cable: sensor (out) to header GPIO4 (same as pin 7)
    3. Ground cable: sensor (-) to header pin 6
  4. Put the case back on the Pi.
  5. Connect the cables to the DHT22, as noted above.
  6. Plug the power back into the Pi and turn on.

Install the Software

Let’s make sure the sensor is working.  In order to use this or any sensor, you’ll need to install software (a python library) that can read and interpret the data coming from the sensor. In this case, we’ll install the Adafruit_DHT library and then write some short Python code to output the sensor data.

From here, I assume you have any version of Python 3 installed.  It’s installed by default, so unless you removed it, it should be a safe assumption.

Start Terminal. By default, this launches a black window with the bash prompt.  At that prompt, tell Python to add the code to read the sensor data. Text is case-sensitive.

  • pip3 install Adafruit_DHT

The script will download and install software and hopefully give you a confirmation everything went well. Techie note for conda users: The library doesn’t appear to be loaded into the Anaconda cloud.  Download and manually install it (like we did with arcgis, if you need the sensor exposed to your conda environment.

Test the Sensor

You need a Python IDE (integrated development environment), a place to write and run code. While most Pis already have at least one Python IDE, you may find it necessary to download one from: Start menu > Preferences > Recommended Software.

Thonny and Geeny were both pre-installed on my Pi and work fine for this task.  Python 3 IDLE is not advisable for this project. With Thonny open, create a new file and copy/paste the Python code below (or download the .py file from here). github

  • import Adafruit_DHT
  • sensor = Adafruit_DHT.DHT22
  • #This is the GPIO Pin number, not just the Pin number.
  • #This pin in pin 7 but sits in GPIO Pin 4. Use 4 below.
  • pin = 4
  • humidity, temperature = Adafruit_DHT.read_retry(sensor, pin)
  • temperature = temperature * 9/5.0 + 32
  • if humidity is not None and temperature is not None:
    • print(‘Temp={0:0.1f}*F  Humidity={1:0.1f}%’.format(temperature, humidity))
  • else:
    • print(‘Failed to get reading. Try again!’)

 

A few notes on the code:

  • Lines starting with # (hashtags) are comments.
  • The indentation on the two print lines is important. Just tab both lines in one time. This is how Python articulates logic and code “flow” (with tabs).
  • Note the import statement at the top.  This is where the Adafruit library is loaded by Python and starts the magic.
  • Most importantly, note the pin= line. This is the GPIO# not the actual pin #.

This sample script is derived from Adafruit’s simpleTest.py script.  I’ve just simplified it a bit but you can find the original here.

If the script runs correctly, the output should look something like:

  • Temp=67.5*F  Humidity=40.1%

Extension Ideas

  • Put a for loop around your code and make it run repeatedly.
  • Put a time.sleep() function in the for loop to space out sampling events.
  • Log your data.
  • Calculate the heat index, based on the temp and humidity data.

 

 

ArcGIS on a Pi: Happy Pi Day!

This post describes how to install the ArcGIS API for Python on a Raspberry Pi 3 B+.
Document version: March 14, 2019. Version 1.1 – Tom Baker RPi-Logo

To learn more about the Raspberry Pi, visit http://raspberrypi.org.

Start with a clean installation of the Raspberry Pi’s Raspbian OS for best results.

Installations

  1. Install Berryconda

    Berryconda for Raspberry Pi is the conda environment for Pi. The installer includes Python 3.6.1 at the time of this writing and meets the minimum Python 3.5 requirement of the ArcGIS API for Python 1.6.0.

    1. Download the bash script (.sh file) for Berryconda3-2.0.0-Linux-armv71.sh from the bottom of the page: https://github.com/jjhelmus/berryconda
    2. With the file on your Pi, right-click and select Properties. Set permissions to “Only Owner” to execute. Press OK.
    3. Open a Terminal window in the current folder and run the bash script. Start the command with a period and paste the filename. Like (ignore indentation icon):
      • ./Berryconda3-2.0.0-Linux-armv71.sh
    4. Click through the Terminal prompts to install. Use default settings.
  2. Install Jupyter Notebook

    1. At a Terminal prompt, type:
      • conda install Jupyter
  3. Install the ArcGIS API for Python

    1. Check your python version. Use the Terminal window and type:
      • python –version
    2. Download the arcgis package from the anaconda repository for 32-bit Linux that matches your version of Python (e.g. 3.6). The file name should look something like: linux-32/arcgis-1.6.0-py36h39e3cac_1.tar.bz2
    3. Download from https://anaconda.org/Esri/arcgis/files
    4. Note: In our tests, the arcgis file cannot be directly downloaded using conda from any repository.The conda tool fails to find the required “linux-armv71” channel for arcgis.  However, downloading the file and installing from a local path negates the conda installer looking for the arm-specific version of arcgis.
    5. Right-click the package and select “Copy Full Path”.
    6. From a Terminal window, install the package like:
      • conda install /fullpath/arcgis-1.6.0-py36h39e3cac_1.tar.bz2
  4. Test the installation

    1. At a Terminal window, type:
      • jupyter-notebook
    2. When the notebook loads in a browser, create a new Python3 document.
    3. In the document’s first cell, enter the four-line test script below that will draw a map.
      • from arcgis.gis import GIS
      • gis=GIS()
      • map1 = gis.map(“Palm Springs, CA”)
      • map1

 

Raspberry Pi Models Tested

To date, we have used the following Pi models with the ArcGIS API for Python and Jupiter Notebooks. Brief comments follow.

Raspberry Pi 3 B+ – Quad-Core BROADCOM 64bit ARMv8 1.2 GHz 1 GB RAM

  • Demo installation above was conducted on this model.  Maps draw responsively with a slight lag on the first request.

Raspberry Pi Zero W v1.1-

  • It took about 5 minutes to run the four line script above.

Mapping Your Geotagged Images

Originally published November 12, 2018 on LinkedIn.

Geotagged images are taken constantly, usually by people with smartphones, perhaps even by people unaware that latitude-longitude information is embedded in the header of the images.  For many casual users, seeing these images in a smartphone’s built-in Photo app with a simple map feature is all the mapping they’ll want.  But for the carto-literati, nothing short of a photo-map service and map will suffice.

This post walks through Python 3 code that uses the Python Imaging Library (PIL) and ArcGIS Online API for Python to extract geotag info, build a CSV, and publish the data out as an ArcGIS Online data service.  The required software stack looks like this:

  1. Python 3.x
  2. Jupyter Notebooks
  3. ArcGIS Online API for Python
  4. PIL
  5. This workflow also assumes your images are stored in a local Dropbox folder(or similar file synchronization tool).

The sample Python code can be found on GitHub.  The sample of the ArcGIS Online map is above and in the article header graphic. After getting the stack installed, you’ll need to download the sample script and enter your:

  1. Local Dropbox folder path
  2. Web URL to the shared Dropbox folder
  3. Your ArcGIS Online details

My sample images included 150+ iPhone images of a recent trip to Italy. The first block of Python code uses PIL to read the image headers and write out a CSV to the root image folder. Because the images are local, this section of the script runs very quickly.  You can stop here if you just want the CSV file or run the second code block to publish the CSV to your ArcGIS Online account. From there, add your new map layer to a map and share! Your markers will be clickable with a link to your images.

Using geotagged images can be a great way to capture verifiable data in a project-based learning or citizen science exercise. Students can collect data with photographs, share their images to a DropBox file request, and then use this script to map the pictures.

Harvesting ArcGIS Online Data and Maps Metadata

This short article describes a process where Python was used to harvest metadata from a list of identified ArcGIS Online maps and the maps’ data services. The data were logged to MySQL (with pymysql); a PHP web search and discovery page was created.  The process allows for keyword searching in titles and descriptions of maps and data layers.

Why harvest metadata?

This approach was used as our collection of 150 maps is housed in several AGO organizations with data services spread across even more.  These data and maps are designed for student use and vetted, making school curriculum authors interested in search and discovery of “good” data.  Essentially, we have a target population that is keenly interested in a subset of scattered data and maps.

Who might use this approach?

This article may be of interest to developers needing to create a search solution across a specific list of maps and constituent data services.

The Approach

Using the ArcGIS Python API 1.5.2 and a prebuilt list of mapIds, a script was built that iterates over the list, logging titles and descriptions for the maps.

with open('data/maps.csv') as f:
reader = csv.reader(f)
for map in reader:
result = gis.content.get(map[0])
web_map_obj = WebMap(result)
web_map_obj
web_map_obj.layers

Using the layers attribute now available for the map object, we then loop over all the layers in the map (above), also logging titles and descriptions.

We then log the data to a MySQL table (with a custom function), like:

dbwriter(
  objectType='Webmap',
  mapId=result.itemid,
  objectName=result.title,
  url='http://www.maps.arcgis.com/home/webmap/viewer.html?webmap=' + result.itemid,
  description=ssnippet
  )

Then loop over each layer and log to the table, slightly changing parameters as necessary.

You can access the script at GitHub below. The Python code, a sample CSV input file, and a sample SQL script for generating the MySQL table are included.

github  https://github.com/trbaker/mapHarvest