Primitive root mod p » Computers

Functional Programming in Python

Nikolas Karalis — Sat, 26 Apr 2008 12:37:31 +0000

Recently, I decided to learn how to use the functional programming tools provided by Python.

Truth is, it’s interestingly strong.

The whole concept consists of using 3 built-in functions (map, filter, reduce), lambda functions and of course the beloved List Comprehensions.

In order of appearance :

map (function, sequence) : It applies function on every item of the sequence.

You can even provide a function which takes two arguments and use it like this :

map (function, seq1, seq2)

filter (function, sequence) : It returns a sequence of the items which have a function (Item) == True value.

reduce (function, sequence) : It applies the function (2 arguments) to the first two items of the sequence, and then to the result and the third etc…

List comprehensions :

This a powerful feature of Python.

I will explain it by an example :

>>> a=range(10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> [x**3 for x in a]
[0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
Lambda functions :

With lambda functions you can create anonymous functions on the fly.

For example, the normal definition of a function is :

def f(n): return n**3

With lambda functions :

f = lambda n: n**3

In both cases, we use like this :

>>> f(5)
125
An example of using lambda functions with the map function would be like this :

>>> a=range(10)

>>> map(lambda x: x**3, a)
[0, 1, 8, 27, 64, 125, 216, 343, 512, 729

This is exactly the same as the example in list comprehensions.

So, with all the above we have a whole new range of tools for dealing with lists.

Seismology, Snow & Procrastination

Nikolas Karalis — Sun, 17 Feb 2008 23:29:19 +0000

I haven’t updated my blog for a while. No specific reason… I was just procrastinating since exams were coming and I didn’t have much to say.

The latest news from Greece are the earthquakes and the snow (and of course my exams ).

The last few days, 2 big earthquakes (6.2 R and 6.9 R) shaked Southern Greece and last night, it began snowing in Athens!

This happens once every one or two years, so it is big news

So last night, a small earthquake reminded me of the passionate discussions about the earthquakes and since i didn’t have much to do because of the snow and my constant last month procrastination, i decided to build a database of the Greek Earthquakes of the last few years.

So, a little bit of coding and a few hours later, I proudly (:P) present my Greek Seismological Database (click here to visit).

EDIT : A new addition : It is automatically updated when a new earthquake happens…

I end this post, with a few photos from Athens in white.

*******

Thesis Database and Python CGI uploading.

Nikolas Karalis — Mon, 28 Jan 2008 16:39:13 +0000

Before a few days I had the idea that it would be really nice if we could have a database of greek theses and dissertations, about mathematics and science in general. From what i know, there are a few databases around, mostly for Electrical Engineering and Computer Science dissertations. So, I thought that it would be a good opportunity for me to exercise my CGI and Python Web scripting skills.

And here I am, presenting the Thesis Database project. I hope that it will be useful and people will contribute.

But while coding the CGI backbone, I had a few problems to solve, so since i had to come up with the solutions (couldn’t find anything useful online), i decided to post them here, for future reference. I will also give the basic idea of how a python cgi uploading script works. The focus is on security of the code.

So, the following is a very simple html form, which will be used as the user interface for the upload.

We suppose that the cgi script is called upload.py and is placed inside the $Web root$/cgi-bin/ directory.

upload.html

Upload Example

File :

upload.py

import cgi, sys, os

try :

form = cgi.FieldStorage() #parse query

fileitem = form["file"]

filename=os.path.join(‘/tmp’,os.path.basename(fileitem.filename)) # We suppose we save it in the /tmp directory (should set the right permissions in the folder) and we manipulate the filename so that we avoid directory traversal attacks.

if os.access(filename,0) : # Check if the file already exists. Important in order to secure our database.

print ‘Filename already exists’

sys.exit()

fout = file(filename, ‘wb’) # We write the file, in parts of 100 Kb. This is used to make sure we can handle larger file, without using all the memory and crashing the system.

for i in range(103): # This is used to set the upload file size limit. 103 is for about 10 MB upload limit.

part = fileitem.file.read(100000)
if not part: break
fout.write (part)
if fileitem.file.read(100000): # If the file is larger than the limit, we remove the already uploaded part of the file.

print ‘The file you tried to upload is too large.’

os.remove(filename)
sys.exit()

fout.close()
print ‘You have successfully uploaded the file.’

except: print ‘A problem occurred. Please try again.’

Comments :

1) The important (non trivial) parts of this code, is the file size checking and handling and the checking for the existence of the file before editing/deleting it.

2) Whenever you see print above, in the string you should include html code, which i didn’t do in order to make the code presented here more compact. If you don’t do this, the chances that your program will work with an older browser is much less.

3) It is a good idea to use a general output html template, which will be edited each time you print something.

4) When coding it is very useful to use the following :
import cgitb

cgitb.enable()

This will print the error output in your browser, so you will know what is wrong with your problem, but you should not include it in the final version, since it can give out many of your code’s details.

Introduction to Algorithms (Greek Translation)

Nikolas Karalis — Sun, 13 Jan 2008 17:09:16 +0000

The following post is written in Greek. It is my complains about the Greek translation of the classic book : “Introduction to Algorithms“, T. Cormen et al.

Αυτές τις μέρες και με αφορμή κάποιες ασκήσεις Αλγορίθμων και γραφημάτων, σκέφτηκα να εκμεταλευτώ την ελληνική μετάφραση του κλασικού βιβλίου Αλγορίθμων των Cormen, Leiserson, Rivest, Stein απο MIT Press, που εκδόθηκε πέρυσι στην Ελλάδα απο τις Πανεπιστημιακές Εκδόσεις Κρήτης (ΠΕΚ).

Οι ΠΕΚ είναι κατα τη γνώμη μου απο τις καλύτερες εκδόσεις στην Ελλάδα, με πλήθος αξιόλογων βιβλίων. Και είχα χαρεί πολύ για την μετάφραση αυτού του πολύ καλού βιβλίου. Αλλά πραγματικά, είναι μια απο τις χειρότερες μεταφράσεις βιβλίων που έχω διαβάσει ποτέ.

Όπως αναφέρεται στον πρόλογο του επιμελήτη :

“Μία απο τις κύριες μέριμνες των ΠΕΚ είναι η προσεκτική χρήση της ελληνικής γλώσσας… Ένα απο τα σημαντικά προβλήματα που έπρεπε να επιλυθούν στη συγκεκριμένη έκδοση ήταν η εύστοχη απόδοση στα ελληνικά της σχετικής επιστημονικής ορολογίας, και η συνεπής τήρηση των όποιων ορολογικών επιλογών καθ’ όλη την υπο έκδοση σειρά.”

Και τα κατάφεραν. Μετέφρασαν κάθε αγγλική λέξη στα ελληνικά, και μάλιστα σε λέξεις που προσωπικά δεν έχω ξανακούσει να χρησιμοποιούνται. Και γίνεται σε τέτοια έκταση που μοιάζει με αυτόματη μετάφραση του Google. Φυσικά δεν κρίνω την ορθή χρήση

των ελληνικών απο τον μεταφραστή αλλά την υπερβολική και (κατά τη γνώμη μου) ενοχλητική και λανθασμένη μετάφραση των επιστημονικών όρων. Πραγματικά λυπάμαι τον άνθρωπο που θα δοκιμάσει να χρησιμοποιήσει αυτό το βιβλίο για να μάθει αλγόριθμους. Προσωπική μου εκτίμηση είναι οτι καλύτερα να αγοράσει κάποιος την αγγλική έκδοση (η οποία κοστιζει 55 ευρώ) παρά την ελληνική (της οποίας ο πρώτος τόμος μόνο κοστίζει 40).

Δεν θα έμπαινα στον κόπο να γράψω για αυτό το θέμα, αλλά κάθε φορά που ανοίγω αυτό το βιβλίο, με πιάνει πονοκέφαλος απο την ακατανόητη μετάφραση.

Python Hacking

Nikolas Karalis — Mon, 07 Jan 2008 03:43:09 +0000

Efficiency Tips

In this post, i will try to collect tips and tricks for efficient Python programming. Feel free to contribute.

I will update every once in a while, so check back…

Fastest string conversion :

>>> a=123

>>> `a`

'123'

Fastest string concatenation :

>>> ''.join['a','b','c']

Efficiency comparison

Optimization Anecdote (Guido van Rossum)

Generator Functions

>>> a=[1,2,3]

>>> b=[x**2 for x in a]

b=[1,4,9]

Timing a process :

from time import clock

t=clock()

...

t=clock()-t

100% CPU Usage Simple Script :

while True : pass

Hash and MD5

>>> a='123'
>>> hash(a)
1911471187

>>> md5(a).digest ()

' ,\xb9b\xacY\x07[\x96K\x07\x15-#Kp'

Code Optimization

import psyco

..

psyco.full()

Compact Code

If possible, try to define less functions, and keep their code in your main function. This of course, reduces code readability and reusability, but it can sometimes (and in extreme cases) give you a slight performance gain. In addition to that, try not to use to many “dots”.

e.g. : Instead of using :

>>>'123'.replace('1','2')

better use :

>>> rep=str.replace

>>> rep('123','1','2')

Urllib

There is a very easy way to open and post data to a URL.

>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
>>> print f.read()

File handling

The easiest way to open a file.

>>> f=open('testfile.txt')
>>> f=open('testfile.txt','w') #Open a file to write.

Useful Links

Compiled Python

Sage

NetworkX

PIL

Pygame

Psyco

Win32 Extensions

Matplotlib

Numpy

SciPy

VideoCapture

Swig

ONLamp Scientific Python

MCI Interface

ReportLab

PyPdf

Deleting duplicate lines from file

Nikolas Karalis — Sat, 05 Jan 2008 20:27:11 +0000

I’ve been fighting with a computational problem for many days, but i haven’t even come close to an acceptable solution.

The problem statement :

Given a file of n lines, return the index number of the duplicate lines (where index line is 0 for the first line, 1 for the 2nd etc.)

It may sound trivial, even silly, but i can assure you it is not. It can be relatively simple for small number of lines.

But as n raises, the problem becomes exponentially harder.

When i first faced the problem, i had to deal with a few hundred thousands of lines. So, i came up with a simple python code to do it.

When the first difficulties appeared, i came up with a faster solution.

  def duplicates(sequence):
	visited={}
	dupl=[]
	for x in sequence:
		if x in visited: dupl.append(x)
		visited[x]=1
	return dupl

However, now that i have to find the duplicate lines in files with 15 million. 26 million and more, it is impossible to use this code, since it returns memory errors.

So i found another idea, which is REALLY slow for now.

Sort the file with the windows or unix command : sort
Use unix command uniq -d, to get a list of the duplicated lines.
Use unix command grep -n, on the unsorted file for every line in the previous list, to get which lines are duplicated.
Use a simple python script to parse the result and get only the the integers we are interested in.

However, the grep part is REALLY slow for huge files. So, my problem remains. However, i reduced the problem from removing duplicated from a file, to simply getting fast the index of a line in a file or equivalently, fast iteration and comparison of the file lines.

After extended digging in the Internet, i was not able to find any efficient algorithm or implementation.

OpenBSD & Security

Nikolas Karalis — Sat, 05 Jan 2008 19:36:57 +0000

I’ve spent the last few days setting up a new OpenBSD server, focusing on the security aspects.

I’m not a security expert, but during this days I’ve learned a few things which i would like to share.

So, I will describe these security tips, without getting into much detail, from general to special. For more details on these, just google or ask me.

Firstly, why OpenBSD?

Well, i had no experience with this OS, so i thought to give it a try. Furthermore, OpenBSD is (generally?) considered to be (one of) the most secure systems around. So, why not?

1) When connecting remotely to your server, always do that with SSH2.

2) You should not login over SSH as root. Create an account, add it to the wheel group so that you can
su with it, and change the PermitRootLogin option to no in /etc/ssh/ssh_config .

3) Use long passwords. You can even set the min. password length as you wish.

4) Always use the latest release, and always patch as soon as possible your system, when a patch is available.

5) Enable only the needed services, and check thoroughly for open ports. e.g. Disable the inetd and sendmail f you don’t need them.

6) After you finish your configurations, change your security level to 2. This is the highest possible, but be aware that this can make administration more difficult.

7) Change your web server’s directives, so that you don’t show to people which server you use.

8 ) Enable your swap partition encryption (this is enabled by default).

9) Use pf (packet filter) to create strict rules for your firewall. For example, block all incoming traffic, and only enable the few services that you really need. In addition to that, use antispoof and packet normalization (scrub) rules.

10) In your pf rules, include rules against bruteforcing. With this, you can avoid DoS (Denial of Service) attacks.

11) Block incoming traffic based on ip, for example allow ssh logins only from your local network or what fits your needs.

12) Try to fool nmap (using pf), so that you not reveal which of your ports are open, and which OS you are using and also log the scanning attempts.

13) Change the sysctl.conf flags, so that your system reboots automatically during a kernel panic.

14) Use file flags like schg, sappnd and uappnd to disable deletion and editing of specific files and directories, like /var/log.

15) Keep a track of your files’ checksums, and your downloads’ checksums.

16) Use snort (or other NIDS and Host IDS) in addition to the pf and system logging to prevent and identify security breaches and attempted attacks.

17) You can use honeyd to create virtual hosts and honey traps for the script kiddies.

18) Use SaDoor for remote stealth administration.

19) If possible (if you are administering the router used), use static ARP in your network, to avoid ARP poisoning attacks.

Facebook!

Nikolas Karalis — Sat, 08 Sep 2007 16:40:18 +0000

The last few months, facebook is open for everyone and not only university students.

I am generally opposed to these websites, but Facebook is one of a kind.
It is aesthetically nice, easy to use and extremely popular (at least in North America).

I tried it, and now i’m using it all the time.

I found so many friends of mine from around the world, that we hadn’t talked in so many years…

So, Facebook me!

Here, is a link to an interesting article regarding Facebook :

Facebook’s plan to hook up the world

Learn Genetics

Nikolas Karalis — Tue, 12 Jun 2007 23:03:39 +0000

I just found a great website… It is hosted at University of Utah.

It has many entertaining and educative Flash presentations of how Genetics works.

http://learn.genetics.utah.edu/

Special thanks to primelude for sending me one of these files…