Wednesday, January 31, 2018

Getting started with AWS Free Tier

Introduction

Please note that i am a beginner with AWS, and this blog is my current understanding, and may not be totally accurate.

Knowledge of AWS has become sort of mandatory these days, so i created an account and will get to try out some facilities free for an year( within usage limits), whereas others ( the "Non-expiring Offers" ) are free for lifetime( again within limits). See https://aws.amazon.com/free/ for details.



For example, EC2 free-tier gives us 750 hours per month on a t2.micro instance. The hours are sufficient for one instance to run continuously per month. If we want to run multiple instances, the hours will have to be divided between them.
So its a good practice to stop your instances after you have finished your practice session with them, to save on hours.

Billing

Billing with AWS has quite complex clauses, e.g. transfer of data, transfer outside a region, transfer over public I.P, non-usage of elastic I.P.s, number of IO reads, etc. Also, the policy is never to stop execution when you exceed the limit, but to charge you. Thus, its better to subscribe to billing alerts at https://console.aws.amazon.com/billing/home?region=us-east-1#/preference
( region=us-east-1 will change as per your settings ) and get notified in time.

Programmatic access with APIs

In order to access your account programatically using Amazon APIs, or tools that use the API, like Boto, Fog, TerraForm, etc, you will to get the access keys from the Security credentials page : https://console.aws.amazon.com/iam/home?region=us-east-1#/security_credential( region=us-east-1 will change as per your settings ). These keys are for the amazon account, and not per instance or service.

About Terraform, its a tool to initialize/destroy instances, but not for installing/updating/running software. Other tools like Ansible,Chef,Puppet should be installed using Terraform initialize, and later used as needed. Also, Terraform saves and reads states from files, and may not be as suitable as Boto/Fog for running on-the-fly configurations without using a file.
When we start/stop/terminate instances, we do not communicate with the instance itself, but rather its region-level handler. This should be clear for start/create, since the instance does not exist. Each instance has an instance-id, which can be used to stop/terminate an instance. We do not need/use the DNS/I.P address.

Storage services

All of these are free only for the trail period. Charges usually apply on amount of data stored, as well as transferred.

S3( Simple Storage Service)

Grow/shrink as needed storage. Not for heavy writes.
Not a file-system with files, inodes, permissions etc. Accessible using its API.
S3 stores objects(files) up to 5 TB, each can have 2 KB of metadata. Each objects has a key, and is stored in a bucket, which itself has a name/id. So its rather like a nested hashmap. Buckets and objects can be created, listed, and retrieved using either REST or SOAP. Objects can be indexed and queried using the metadata/tags. Can be queried as SQL using the Athena service. Can be downloaded using the HTTP or BitTorrent. Bucket names and keys are chosen so that objects are addressable using HTTP URLs:
  • http://s3.amazonaws.com/bucket/key
  • http://bucket.s3.amazonaws.com/key
  • http://bucket/key (where bucket is a DNS CNAME record pointing to bucket.s3.amazonaws.com)
Because objects/files are accessible via HTTP, S3 can be used to host static websites. Some dynamic scripting could be provided by Lambda.
S3 can be used as a file-system for Hadoop.
Amazon Machine Images (AMIs) which are used in the Elastic Compute Cloud (EC2) can be exported to S3 as bundles.
https://aws.amazon.com/blogs/aws/amazon-athena-interactive-sql-queries-for-data-in-amazon-s3/

I just discovered that creating tags on S3 objects incurs costs, tho quite small ! AWS billing is really tricky. Fortunately, due to the billing alarm set up, i got notified in time.

EBS( Elastic Block Storage )

Fixed amount of block storage for high throughput. (Fast read/writes ) e.g. can store DB files. Multiple such allocations can be made. Needs to be attached to a file-system. Should be formatted before using. Attached and accessible only to a single EC2 instance.

EFS( Elastic File System )

Grow/shrink as needed, managed file-system, can be shared among multiple EC2 instances. Not HTTP accessible, no meta-data querying like S3.

Glacier

Cheap, long term, read-only archival storage

Serverless services( Lambda) :

Serverless means one does not have to setup servers or load-balancers. We just write a function that does the required processing. e.g store price updates into DB. The server, scaling is all handled by AWS. The charging is only for the use, not for the uptime. So if the functionality is not called frequently, one could use Lambda instead of an always-on EC2 instance, and be billed less. The function can be called from various sources like S3 object modifications, logging events, or an API Gateway interface that accepts HTTP calls. Using these sources may incur separate charges.

Working with EC2(Elastic Computing) instances

Its quite easy using the management console to launch an EC2 instance. The options that are eligible for the free tier are marked as such, e.g."t2.micro" instances. Sometimes, options that may incur additional charges are marked as such.

Regions

The region is important since the console usually lists instances/services for the current region. Also communication between different regions may incur additional charges. The APIs too usually query by region. So for testing, its better to keep everything under a single region. In real life, distributing your application across different regions will provide better fail-safety.

Tags

Its possible to add tags when configuring an instance. e.g. type=DB. These tags can be used by the API, e.g. to filter out only DB servers and work on them.

User-data

Specify a set of commands to run when the instance starts, e.g. set env variables, starts a DB daemon, etc. If using config/script files, where will be files come from ? Probably from an S3 or EFS storage that will have to be set up first. This option is available under "Configure Instance Data->Advanced Details". The user-data runs only the first time the instance is started, if we want it to run on every restart, use the "#cloud-boothook" directive at the start of the script.
Here is an example of setting env-variables, and copying and running a file from S3:
--------------
#cloud-boothook
#!/bin/bash
echo export MYTYPE=APPSRV > ~/myconfiguration.sh
chmod +x ~/myconfiguration.sh
sudo cp ~/myconfiguration.sh /etc/profile.d/myconfiguration.sh

aws s3api get-object --bucket <bucketname> --key test.sh /home/ec2-user/test.sh
sh test.sh
----------------

The EC2 parameter store seems to be another way to store parameters, with better security

Addresses

When an instance is created, it is assigned public and private I.Ps, as well as a public domain name. The domain name is of the form
ec2-<public I.P>.compute-1.amazonaws.com.
If we use the domain-name from outside AWS, e.g. from our local m/c, it will resolve to the public I.P of our instance. If used from within AWS, i.e. from an EC2 instance, it will resolve to the private I.P of our instance. The domain-name usually contains the public I.P address, and changes if the public I.P changes. The public I.P is not constant, as its allocated from a pool. A reboot does not change the I.P.s. However, a stop and start will change the public I.P, tho not the private I.P. One solution for a fixed public I.P is to use elastic I.P.s. However, they can incur charges in certain cases, e.g. if not used. A terminate and create-new will of course change the I.P.s.

For better security, it should also be possible to have only a private I.P. for some EC2 instances,and access them via SSH from the EC2 instances that have a public I.P. This is probably the "Auto-assign Public IP"option, enabled by default

Key-Pairs

We usually work on an instance using SSH, with public/private keys. (These are different from the API Access keys for the account.) These key-pairs can be generated from the console, and associated with an instance. ( Advanced : Can also be generated on your local m/c, and the public key copied to the proper directory on your instance ). Has to be done when creating an instance. If using the Launch wizard, you will be prompted for creating a key-pair, or using an existing one. A key-pair can be shared among multiple EC2 instances. Make sure to use a name that keeps the file unique on your local file system.e.g. use aws, account etc in the name.

Security Groups

Each EC2 instance is associated with a security group, which is like a firewall. It controls what protocols and ports are available for inbound and outbound calls.

IAM Roles

For internal use within AWS services. e.g. Accessing S3 from EC2 requires the  account secret keys for auth. Instead one can create an IAM Role with S3 permissions, and grant to EC2 instance. Not very flexible though. Can be specified only when launching the instance. Also, combination of roles cannot be granted.

Storage

The instance launch wizard  will by default create an 8 GB EBS root volume for the instance. In addition there is an option to attach more volumes. For free-tier, only EBS seems to be supported. There is a "Delete on termination" option, which if checked, will delete the EBS volume after the instance is terminated. Stopping an instance won't affect the EBS volume tho, and i checked that some files that i had added to the volume were intact after a Stop and Start.

Monday, October 3, 2016

A collection and player for Indian Classical Music Bandish

Some years ago, around 2010, i had added an "Indian Classical Music" section to my site  http://milunsagle.in
I used ascii letters and symbols for the notation. e.g. SrRgGmMPdDnN.
A full notation help is available here : http://milunsagle.in/webroot/pages/bnd_notn_help
The java midi api was used to create midi files for the bandish( compositions) and play them.
However, with the restrictions on applets and Java Web start in browsers for security reasons, support for the java player in browsers became more and more difficult.
Now unlike reading/writing local files or executing a program, outputting music is not usually a security concern.
The relatively recent web audio api, addressed this issue and made playing audio thru JavaScript possible. There are many frameworks like Tone.js, which allow us to play tones thru javascript. I have moved the bandish-payer on my site for indian classical music from applets to Tone.js. Currently meend(glide) and andolan, which were covered in the java version, are not yet available.
The "Play" link will invoke the player. Here is a link to a bandish :
http://milunsagle.in/webroot/bandishes/view/28

Monday, July 4, 2016

Converting a pdf to csv using linux shell script

linux script to extract data from pdf and create a csv. The regular expressions for sed are rather different from the Perl like ones i am used to in java. So \d is not allowed, + needs to be escaped, etc.

Below, we iterate thru pdfs, use pdftk to get the uncompressed version that has text, use strings to extract string data, use tr to remove newlines, apply sed on it to extract particular fields that we want, assign those to variables, and echo the variables to a csv file.

rm pdf.csv
for FILE in *.pdf
do
  echo $FILE
  pdftk "$FILE" output - uncompress | strings | grep ")Tj" | tr '\n' ' ' | sed -e 's/)Tj /) /g'  > temptocsv.txt
  AMOUNT=`sed -e 's/.*(Rs \:) \([0-9]\+\).*/\1/' temptocsv.txt`
  CHLDATE=`sed -e 's/.*(Date of) (challan :) (\([^)]\+\)).*/\1/' temptocsv.txt`
  SBIREFNO=`sed -e 's/.*(SBI Ref No. : ) (\([^)]\+\)).*/\1/' temptocsv.txt`
  CHLNO=`sed -e 's/.*(Challan) (No) (CIN) \(.*\) (Date of).*/\1/' temptocsv.txt`
  echo $FILE,$CHLDATE,$SBIREFNO,$CHLNO,$AMOUNT >> pdf.csv
done

Saturday, March 28, 2015

USB HID based IO utils

This project was started with the idea of turning a USB device into a kind of IO Swiss-knife, so that many things could be done with a single program. The use-case was to implement a remote-control, first to read the durations, and next to write them back. I had success in the read, by adjusting for code-delays. (The writes require more accurate timings, and i wrote a separate app for that. See  chaukasalshi.blogspot.in/2015/03/pic-micro-remote-for-tata-sky.html ) This project is based on Microchips Custom HID MLA project. Not all files are provided, only the main ones that were changed. The app_device_hid_custom.* files replaced with app_device_hid_io.*. Actually, i wanted to abstract the io routines into pin_io module, which would be independent of protocol, i.e USB/Serial/Other. And to create it as a standalone project with as few files as possible. Some such items are TODO, i may not get time to finish them. i have used the 18F4550 for this project. Probably, the project is quite primitive, considering the level of expertise on these forums, but i hope it may help somebody to begin.
The project files are available here : github.com/manojmo/pic_micro/tree/master/hid_io

Firing the IO commands :

This is done using the pyusb framework. The hid_io_test.py file has the code to send the commands and receive the results. Just as a POC, the results can be saved to a .JS file, and viewed as a chart in mychart.html, using the Chart.js framework. That needs to be downloaded separately. The script also has a facility to adjust results, based on expected timing inaccuracies. For the sampling commands, num_samples are specified, and the script will calculate the number of packets and loop to receive them.

Common features

The framework is generic, and the commands accept input like which pin to use, what delay intervals, etc. (As a result, accurate timing is an issue. ). There is a debug feature that can return you the actual time taken for an interval. (not very exact) Routines are provided to measure and execute 2^24 cycle delays. Timings can be specified in uS or mS. There are flags for each command, to provide additional control. All sampling commands also specify an IDLE state/value, and sampling does to start until the input changes from the IDLE state/value. This is useful for manually-triggered sampling, like reading from a remote.
The commands are

WP : Write Pin :

This allows us to write a pattern to the <pin_num>. A pattern is an array of 2-byte durations, starting from HI. Currently, upto ~ 60 durations are supported. They end with a zero duration marker. There is a facility to make the HIGHs pulsed at a carrier-freq, as needed for IR codes. A reset flag, mS/uS flag, and a repeat flag are also provided.

RP : Read Pin :

This allows us to read <num_samples> of HIGH/LOW states from <pin_num> at <sampleInterval> intervals. Not well tested. TODO : store the result in bits instead of bytes. A debug flag and mS/uS flag are also provided.

RPD : Read Pin durations :

This allows us to read <num_samples> durations of HIGH/LOWs by sampling <pin_num> at intervals of <sampleInterval>. durations upto FFFF cycles are currently supported. A debug flag, idle-state flag and mS/uS flag are also provided. This was used to read durations of IR-code from tata-sky remote, but i needed to adjust the values for code-exec delays. i have blogged about it here : chaukasalshi.blogspot.in/2015/03/pic-micro-remote-for-tata-sky.html

RA : Read ADC :

This allows us to read <num_samples> samples from ADC at intervals of <sampleInterval>. A debug flag, idle-value and mS/uS flag are also provided. The results can be viewed as a chart.

Friday, March 13, 2015

PIC micro remote for tata sky

Background

My entry into the world of micro-controllers was out of my desire to create a USB device. I wanted to create a general purpose IO framework that could be used to do many useful things, and one of my use cases was to add infra-red capability to my mobile which had none, and use it like a TV remote. While that framework is under way, and i may blog about it later, what i realised was that the runtime overheads of generic code might prevent me from achieving the exact timings that are required for remote protocols like RC6. Hence i fell back to writing a more basic/specific code for this purpose.

I have tested this code on the PIC18F4550.

Basics

My work would be in two phases, first find the codes, and second, transmit them. Since i did not find any reference for codes for the tata sky settop box, i had to find out the codes myself. i.e. what was the tata-sky remote transmitting. My starting idea was that it would be a series of ON & OFFs, with specific durations. To capture them, i would need an infra red sensor.  i searched the net and found the following links : 

This is a project for arduino, and describes how to capture the IR codes for a Nikon camera, and how to play them back. Since i was programming with PICs, this project would not work as is for me and i had to start on my own code.

http://www.sbprojects.com/knowledge/ir/index.php is also a good reference for knowledge of IR protocols. 

One basic thing to understand, is that the HIGHs in the code are usually transmitted as a series of pulses at high freq, called the carrier frequency. This is so that ambient light/disturbances are not mistaken for a code. Also, it lets the IR diode cool, since it takes rather high current, 50- 100mA. 

There are 2 kinds of sensors, a simple IR sensitive transistor/diode that would capture the signal as is, i.e. even the carrier pulses. This is available for 4-5 Rs. The other filters out the carrier pulses, so we directly get the highs and lows of the code, it costs slightly more, around 25 Rs. i got both, and experimented with them. The plain receiver has low sensitivity, and also, i did not have a oscilloscope to measure the waveform exactly. The one with the filter had good sensitivity so i decided to go with it. 
The one i got was TSOP1730( carrier freq 30Khz). Note that its output is inverted, i.e. it is usually HIGH and when it receives IR input, it goes low.
i also got an IR diode for transmitting for 4-5 Rs.

Trial and error

i added a facility to the read code to define an IDLE state and wait until it changes. so we can run the program, it will wait till input is in IDLE state, in our case, HIGH. when we press the remote, the state changes and now it starts taking the reading, i.e. measuring the durations of ones and zeroes. 

Why measure durations, why not just measure the sequence of 1s and 0s ? The reason being that in protocols like RC5/6, both 0s and 1s are represented using both the HIGH and LOW states.  HIGH first, LOW next could mean 1 , and LOW first, HIGH next could mean 0 etc. The durations are important and need to match the protocol.

One of the problems with the TSOP1730 sensor is that it can get randomly triggered by surrounding disturbances, so sometimes the program would start taking readings without my pressing the remote. Its suggested to tie the TSOP output pin to VDD using a > 10K resistor, and putting a capacitor betw the TSOP's VDD and GND to reduce power-supply interference. i also tried to tie the output to GND instead, thru a 175K resistor, and used a .1uF capacitor betw the TSOP's VDD and GND. None of the methods proved totally fool-proof however, and the random triggering still happens sometimes. 

i started to look at the readings i was getting on pressing the remote. then i would try to transmit those durations, inverted now, (since out sensor was inverted ), and see if it worked, but no luck. i did not even know what protocol my remote used, so i had no idea if my readings where right or not.

i searched the net again, came up with link link : http://forum.arduino.cc/index.php?topic=166536.0, that somebody else had built a remote for tata-sky and the protocol was like RC6, tho no details were provided. The guy had built it from the arduino link i have given at the start, so i started looking at it again. one thing i realised was that timing need to be accurate. 

at this time, i was using a generic io framework, where one could specify in the request which pin to use, what carrier-freq, the durations etc. i started measuring the timings of my transmit and read using the PIC's timer, and realised that the generic nature of the framework was adding a lot of overhead over direct hardcoding. So i changed the generic code to bare and specific code for the task in hand. One more problem was that my readings did not fit the RC6 protocol exactly; they extended some bits longer. i decided to try with both options, i.e. once stopping at RC6 length, and once with my readings as is.
i started to test again, however, still no success. The settop box links red when it receives a valid code, and this was not happening. i went back to the article to check what else i was missing, and there it was. The article said that we need to send the code twice, not once. i added the repeat with a gap of 65ms, and the set-top box finally responded by changing the channel. yay ! It turned out that the readings that i had got were correct, even tho longer than the RC6 length.
The strange part was that my readings did not show the code being repeated, so how did the remote work ? maybe the remote encodes some more info that i have missed.

The timings of my readings were another issue, they did not match RC6 timings, where one unit is of 444 uS. So i wrote a python script to change my readings to RC6.


def fit_to_rc6( durations) :
    rc6durations = []
    for loop in range( 0,len(durations) ):
        duration = durations[loop]
        newduration = duration
        if  duration < 600 :
            newduration = 444
        elif duration < 1000 :
            newduration = 889
        elif duration > 1800 and duration < 3500 :
            newduration = 2666
        rc6durations.append( newduration)
    return rc6durations


arr_durations = fit_to_rc6(
[3015,474,435,400,408,400,408,852,408,839,855,407,408,407,408,413,408,407,408,393,428,407,408,413,408,407,408,413,408,407,408,413,408,407,408,407,968,744,848,407,408,400,408,852,408,413,402]
) 
 
Here is a sample of the readings i got :
the program has a max wait of  65535 us. after the signal ends,we get all max-durations i.e. 65535. Also the sensor gives inverted output,so S=0 is actually HIGH input to the sensor
Starting..                      S:0,D:2662,S:1,D:656,S:0,D:495,S:1,D:265,S:0,D:528,S:1,D:224,S:0,D:535,S:1,D:663,S:0,D:522,S:1,D:656,S:0,D:948,S:1,D:218,S:0,D:542,S:1,D
:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1
,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:955,S:1,D:656,S:0,D:522,S:1,D:231,S:0,D:535,S
:1,D:224,S:0,D:955,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:6
5535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:655
35,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535
,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S
:1,D:65535,S:1,D:65535,Done..

we can see that the reading vary quite a bit, from 224 to 542 are all actually 444 us, and from 656 to 948 are all 889 us ? how do i know ? by getting a bunch of readings, and seeing which part repeats most, and comparing with the Rc6 protocol. The remote_codes.txt file contains codes for some common keys.
Most of them have been implemented in the sample code too.

about the transmitter, its a simple transistor circuit, with the base driven from the PIC, and the IR diode with a 100R resistor in series with the collector. emitter to ground.

Code

The code is available at https://github.com/manojmo/pic_micro. The read_pin_durations.c is for the reading of the remote codes. The tata_sky_remote_serial.c is for transmitting the codes. it uses a uart to accept a code, and looks it up from an array and transmits it. The array contains the durations of ON/OFFs. Ideally, since we know the protocol and timings, we could use a more compact format to store the codes, rather than actual durations. e.g, we could keep the start-bit of 2666 and 889 separate, and represent the rest with 444 as 0 and 889 as one, and hold the entire code in just an int. Also, no keyboard interfacing is done. One could use a multiplexer, to get 16 lines from 4 pins, or implement it in the PIC. a good link to understand a keyboard is
http://pcbheaven.com/wikipages/How_Key_Matrices_Works/

Other references

Ken sheriff has a IR library for Arduino, which can capture and play-back remote code on the go ! cool, no ? http://www.righto.com/2009/08/multi-protocol-infrared-remote-library.html

Another project for PICs is http://www.compendiumarcana.com/irwidget/

Monday, March 2, 2015

Generic UART for any microcontroller

Communicating with a micro-controller at runtime, is a very useful facility. It can be used for debugging, or to send commands etc. One of the easiest available protocol to communicate is UART. It requires a pin for transmit and a pin for receive, and the GROUND connection. There are many available programs that allow us to communicate using UART on the serial port, or using a usb-serial adapter. e.g. hyperterminal, minicom, etc.

However, not all microcontrollers have UART ports in built.

i was taking a look at the UART protocol, and it seemed pretty simple to implement.  So i gave it a go, and was zapped when it just worked the first time :).

Below is a little app that accepts 2 byte commands. Toggles LATD1 when it receives the command "TG", and echoes back the command after its done. Invert flag is provided. ( Inversion may be needed if we are not working directly with a real serial port). Tested with minicom @9600 baud, with hardware-flow-control OFF, so that it sends the chars we type.

It is tested on the PIC18F4550, will need modifications to run on other microcontrollers.

Its also available as a module to include at https://github.com/manojmo/pic_micro


/*
 * File:   main.c
 * Author: manoj
 * Writing to uart
 * Created on September 27, 2014, 6:01 PM
 */

#include 
#include 
#include 

// PLL with pre-scaler and post-scale options is a way to derive multiple
// freq from the same source, e.g. for low=6Mhz/high=48Mhz speed USB and for MCU clock
// Input to PLL has to be 4 Mhz and its output is 96 MHz
// So, for e.g. if we are using exernal 20 MHz osc, its o/p has to be
// divided by 5 to get 4 Mhz input for PLL
// PLLDIV is prescaler, o/p has to be 4MHz
// CPUDIV and USBDIV are postscalers, input is 96Mhz

#pragma config PLLDIV   = 5         // (20 MHz crystal on PICDEM FS USB board)
#pragma config CPUDIV   = OSC1_PLL2
#pragma config USBDIV   = 2         // Clock source from 96MHz PLL/2
#pragma config FOSC     = HSPLL_HS

#pragma config IESO = OFF
#pragma config WDT = OFF
#pragma config STVREN = ON
#pragma config LVP = ON
#pragma config BOR = ON
#pragma config MCLRE = ON
#pragma config PWRT = OFF
#pragma config PBADEN = OFF

// Effective CPU Freq, considering PLL and CPUDIV. Needed for the delay routines
#define _XTAL_FREQ 48000000

#define UART_BIT_TIME 104 // us. inverse gives the bit-rate
#define PIN_UART_TX LATD4 // output
#define PIN_UART_RX PORTDbits.RD5 // input

uint8_t UART_INVERTED = 0; // If inverted, low and high states will be inverted
uint8_t HIGH_STATE = 1;
uint8_t LOW_STATE = 0;

// flexible way to set some params.
void uart_init( uint8_t is_inverted){
    UART_INVERTED = is_inverted;
    if( is_inverted){
        HIGH_STATE = 0;
        LOW_STATE = 1;
    }
}

// start of transmission
void uart_start_tx(){
    PIN_UART_TX = HIGH_STATE;
}

// end of transmission
void uart_end_tx(){
    PIN_UART_TX = HIGH_STATE;
}

// write a char
void uart_write( char c){

    // start bit
    PIN_UART_TX = LOW_STATE;
    __delay_us( UART_BIT_TIME);

    uint8_t bit_pos = 0;
    // transmit bits, LSB first
    while (bit_pos < 8) {
        uint8_t currbit =  c & 0x01;
        PIN_UART_TX = UART_INVERTED ? ! currbit : currbit;
        __delay_us( UART_BIT_TIME);
        bit_pos++;
        c = c >> 1;
    }
    // stop bit
    PIN_UART_TX = HIGH_STATE;
    __delay_us( UART_BIT_TIME);
}

// Read specified number of chars and put them into holder array
void uart_read( uint8_t len, char holder[] ){
    uint8_t i;
    uint8_t bit_pos;

    // loop and read len number of chars.
    for( i=0; i< len; i++){

        // Wait for idle state to change, i.e. start bit
        while( PIN_UART_RX == HIGH_STATE);

        // start bit
        __delay_us( UART_BIT_TIME);

        // read bits, LSB first
        bit_pos = 0;
        char c = 0;
        while (bit_pos < 8) {
            uint8_t currbit = UART_INVERTED ? ! PIN_UART_RX : PIN_UART_RX;
            c = c | (currbit << bit_pos);
            __delay_us( UART_BIT_TIME);
            bit_pos++;
        }
        holder[i] = c;


        // stop bit
        __delay_us( UART_BIT_TIME);
    }
}

// write a string, optionally of specified len.
// if null terminated, len can be -1
void uart_writestr( char str[], int len ){
    int i;
    for( i=0; i< len || len == -1; i++){
        char curr = str[i];
        if( curr == '\0'){
            break;
        }
        uart_write( curr);
    }
}

// process the command we received
void process_cmd(char cmd[] ){
    if( strncmp( cmd, "TG", 2) == 0){
        LATD1 ^= 1;
    }
}

/*
 *
 */
int main(int argc, char** argv)
{
    TRISDbits.RD4 = 0; //TX pin set as output
    TRISDbits.RD5 = 1; //RX pin set as input
    TRISDbits.RD1 = 0; // command output

    uart_init(1); // invert
    uart_start_tx();
    char cmd[2]; // array to hold received command.
    while(1){
       uart_read( 2, cmd); // read a command, specified bytes long
       process_cmd( cmd);
       uart_writestr( cmd, 2); // echo it back
    }
}

Monday, January 19, 2015

Controlling Microchips 18F4550 as HID Custom USB device using python pyUSB

The MLA( Microchip Libraries for Applications ) ships with a MPLABX project for 18F4550 that makes the chip behave like a HID Custom device. i built the project, burnt the hex file, and was also able to test it using the plug_and_play executable provided for Linux

As per this project, there is 1 interface with 2 endpoints.
One is IN, other is OUT. Both are of type INTERRUPT.
( See usb_descriptors.c )

For the commands, see app_device_custom_hid.c.
The commands are :
  • COMMAND_TOGGLE_LED = 0x80 -- LED on RD0
  • COMMAND_GET_BUTTON_STATUS = 0x81 -- Button on RB4, pressing should bring it to Ground
  • COMMAND_READ_POTENTIOMETER = 0x37 -- ADC_CHANNEL_0

Commands should be sent on the OUT endpoint, and data received on the IN.


i wanted to write my own code to control the device. came across the pyusb framework, which looked easy to use.

Here is working code for the first 2 commands :
import usb.core
import usb.util
import sys

# find our device
dev = usb.core.find(idVendor=0x04d8, idProduct=0x003f)

# was it found?
if dev is None:
    raise ValueError('Device not found')

interface = 0

if dev.is_kernel_driver_active(interface) is True:
            print "but we need to detach kernel driver"
            dev.detach_kernel_driver(interface)

# set the active configuration. With no arguments, the first
# configuration will be the active one
dev.set_configuration()

# get an endpoint instance
cfg = dev.get_active_configuration()
intf = cfg[(0,0)]

for cfg in dev:
    sys.stdout.write("ConfigurationValue:" + str(cfg.bConfigurationValue) + '\n')
    for intf in cfg:
        #print vars(intf)
        sys.stdout.write('\t' + \
                         "InterfaceNumber:" + str(intf.bInterfaceNumber) + \
                         ',' + \
                         "AlternateSetting:" + str(intf.bAlternateSetting) + \
                         '\n')
        for ep in intf:
            #print vars(ep)
            sys.stdout.write('\t\t' + \
                             "EndpointAddress:" + str(ep.bEndpointAddress) + \
                             '\n')

assert ep is not None

# toggle the led. COMMAND_TOGGLE_LED
# args : ep, value, timeout
dev.write( 1, chr(128), 100)

# read the button state. COMMAND_GET_BUTTON_STATUS
# first send the command on EP_OUT
dev.write( 1, chr(129), 100)
# read on EP_IN
ret = dev.read( 129, 100)
print "Received: %d,%d", ret[0], ret[1]
print "Button Status : " + "Released" if ret[1] == 1 else "Pressed"