Custom Linux Software - need help with field sequence in upload data

[shed41]

Hi Folks,

I am scrounging various bits of "C" & "C++" code from all areas of the internet and developing a custom executable software package that I will be using to upload data from a weather station.

The software will sniff the data out of the weather station and I will re format that data and upload it as required ...

Just to be clear, this will be pure linux only.

Further, my Modem (and a lot of others) run internally a Linux operating system ...

Hence me gathering, modifying, writing my own version of software code (( not 100% mine as I am collecting from other coders as well so credit for there bits belongs to them respectively )) which will run as a separate program much the same as any other program runs on any computer running linux or windows with the difference being it will run on the modems internal linux software system.

What I need from you localweather nz guru's is the layout of the separate fields extracted from the weather station to upload to the localweather.net.nz site.

for example when I upload to weather underground I do so similar as follows ...

weatherunderground.com/service.php?action=updateraw&ID=MYID&PW=mypw&dateutc=%n&winddir=%d&windspeedmph=%w&windgustmph=%g&humidity=%H&tempf=%o&dewptf=%e&baromin=%l&indoortempf=%i&indoorhumidity=%h&rainin=%s&dailyrainin=%t&softwaretype=CustomWeather%%20running%%20on%%20Custom%%20Linux

What I need is the nzweather.net.nz php server address to send a metric data set to

So who can help me with that

Incidentally, I am now running a trail setup and uploading to Weather-Underground under my station-id IWAITAKO47 see following link

http://dev.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IWAIKATO47

My newly acquired station unit is an Aercus WS3083

Thanks  John

• Tweet

[gabba]

Hi john

Interesting stuff! For the nzlwn map it uses php script to retrieve / pull updates sitting on individual pcs. I'm fairly sure the daily data that jenneyleez does also uses the same method.

I'm not aware of any push functionality where info is sent via the URL for either of these methods, but I'm sure others will jump in and say so if that's not correct

[JennyLeez]

Hi John,

The Map and the Data summary scripts both read a data txt file uploaded by our stations to their servers.
The server part is easy and I will email the username, password and ftp path to you when needed.
Simply put. Our scripts count along the line of data extracting the values required. In the case of the map, it is the realtime values. In the case of the summary run before midnight, it is the high and low values it counts along and reads.

Beteljuice may possibly have a better understanding than me but, in my mind, what you need to do is a replica of the cumulus realtime.txt.
I suggest cumulus's realtime as it is a quarter the size of weather displays clientraw.txt.
Here is an example.
04/04/14 00:31:11 11.8 98 11.5 3.6 5.0 360 0.0 0.0 1024.7 N 1 km/h C hPa mm 1.5 -0.2 0.0 192.0 0.0 21.5 62 11.8 -0.9 12.2 00:00 11.8 00:26 3.6 00:00 6.1 00:23 1025.0.............

So your software would need a file to process if this is possible.
Substituting the values with the current data. e.g.
$temp,$rainfall,$windspeed,$windgust
I have no idea what you will be using. Could be %rainfall or #rainfall. You will know
Then uploading the processed file every couple of minutes.

I could make a dummy txt file for you. Put in the values we need and put 0.0 for the values we do not need. The 0.0 will be place marks as the scripts counts along for the data it needs as I said above.
e.g. looking at the demo txt line above. 1 = date, 2 = time, 3=current temp, 4=humidity.

Ok I hope the above info is what you are after. Its 12.30am and after a day of viruses, spyware and clients I am brain drained

Let us know what we can do to help further.
Cheers
Jenny

[beteljuice]

To clarify ...
Data used by NZLWF is trawled from a data file on the members server - It is not 'pushed' to NZLWF server.

You need to have a 'realtime' data file in Cumulus or WD format on a web space that NZLWF can access. ('realtime' is very loose and upto the member, typically 5 minutes or less turnaround)

'realtime' also contains some historic data ! - eg. min / max values etc.

The format for Cumulus is here: http://wiki.sandaysoft.com/a/Realtime.txt

Whatever 'format' you use, the datafile would have to be named in that format style. eg. realtime.txt, clientraw.txt.

NZLWF runs routines to trawl the 'donors' sites at specific times / intervals and extracts the relevant data it requires.

Edit: Should you have an 'external site' you can upload your data to / create web pages; as long as you adopt one of the 'recognized' formats you will find a plethora of 'templates', php scripts, javascripts, etc. etc. and you can rule the world 

[shed41]

Mmm going to have a think on the replies so far ...

Bit stunned localweather.ne.nz does not have an active php script running so as to accept uploads in thier required field layout  ...

I might start writing one up so someone can load it up onto the lw site for beta testing.

In the mean time I can confirm that my available output fields are as follows ...

%% - record age
%% - weather station type
%% - wind speed m/s
%% - wind speed km/h
%% - wind speed mph
%% - wind gust km/h
%% - wind gust mph
%% - wind direction named
%% - wind direction degrees
%% - outside C temp
%% - outside F temp
%% - outside humidity
%% - outside C dew temp
%% - outside F dew temp
%% - outside wind chill C temp
%% - outside wind chill F temp
%% - inside C temp
%% - inside F temp
%% - inside humidity
%% - relative hPa pressure
%% - relative inch pressure
%% - absolute hPa pressure
%% - absolute inch pressure
%% - UV radiation
%% - illumination lux
%% - illumination W/m2
%% - rain last mm in 60 min
%% - rain last inch in 60 min
%% - rain mm from 0h local time
%% - rain inch from 0h local time
%% - rain total mm from meteostation start
%% - rain total inch from meteostation start
%% - time string HH:MM local time
%% - date string DD.MM.YYYY local time
%% - date/time string YYYYMMDDhhmm local time
%% - date/time string YYYY-MM-DD HH:MM:SS UTC time
%% - date/time string YYYY-MM-DD HH:MM:SS local time

Some data sniffed and some calculated ...

Being beta and me being mere mortal there will be the odd bees dick worth of errors in data but I'll get there eventually haha

Don't pay any attention to the order as stated here as fields are easily re-arranged to suit any particular purpose ...

At this stage I would like a few lines of output from whatever program puts out the realtime.txt file (cumulus??)

Preferably with the First Line showing the Field Names, and the rest some sample data.

It would be helpful if the field data types were stated also.

From there I should be sweet ;-)

[shed41]

Might ad that I DO NOT have or run any form of windows running I'm 100% linux here so do not have any idea what your windows based software can do or puts out ...

[JennyLeez]

Ok here we go

Here is a dummy realtime.txt
http://wairoa.net/weather/realtime.txt

Here is the field and the number order.
Download the Cumulus Data Log Format Manual Here.
The bottom one is called realtime.txt. You will see the order you need to make your file. If you do not have the value then 0.0 as a place mark.
So looking at your list, %% - date string DD.MM.YYYY needs to be the first one.
The second one %% - time string HH:MM   Can you supply seconds hh:mm:ss

This is what you need to replica.
I will email you a list of the fields that need to be filled.
You will have to fill all fields though as the scripts count them out. Hence why I said up above use 0.0 as a placemark for a field we dont need.
Example. For the Rain rate put 0.0 as we do not need this one.

There is 2 totally different scripts lifting different data from this file. Although the NZ Network uses these scripts, the NZ network does not own them
The map can read 5 different weather software packages.
http://www.localweather.net.nz/smf/nz-local-weather-network-map-requests/weather-station-software-configuration/
The data summary only 2.
So if we set you up for the Data Summary, then the map script will read the same file.

Hopefully beteljuice will check in later and add or correct me

[JennyLeez]

Bit stunned localweather.ne.nz does not have an active php script running so as to accept uploads in thier required field layout  ...

Our stations do not upload to the network. They upload to their own websites. The likes of this cumulus realtime.txt and weather displays clientraw.txt are uploaded to their sites as their websites use them.
Most live sites hinge off these types of files to display their data.

What our scripts are doing is using what our members already have in place. This makes it easy to set them up on the Map and data summary. It also means they are not uploading a duplicate to us of what they are already sending to their own server.

I hope that clarifies that.

[gabba]

You might want to check out the following

http://www.weather-display.com/files.php

A number of us use the windows version if this app, and I see there are some Linux variants, including a pi version. Brian who wrote the software occasionally pops in here so if you are interested that might be an alternative way to go

Providing a script to accept uploads might be a but tricky. We would also have to integrate the data somehow into our existing structure probably by creating the file on the localweather.net.nz server.

I'd have to look again about how the wunderground upload works, but polling files off servers is how all our other stations work.