I solved it now: You have to add parenthesis around the MySQL functions AND you have to separate the parenthesis with spaces!
A working example for a nested filter:
Field: "( starttime"
Operator: "<="
Value: "( UNIX_TIMESTAMP() ) ) OR ( starttime = 0 )"