Both Lewis Hamilton and Valtteri Bottas struggled for pace in mixed conditions in FP1 and again in the dry in FP2 - the pair ending the day sixth and seventh, both more than 1.4s off Sebastian Vettel's leading Ferrari.
Wolff said the team are struggling to understand the downturn, and that they "need to find out tonight what it is".
He added: "There is a gremlin in the car; a fundamental issue. They are complex machines so it can be a tiny detail that just unsettles the car - but we need to find it.
"This is a very different track to Singapore, but you can maybe compare it to Silverstone where we were very good in the high-speed sections. We're not here.
"We struggled from the get-go this morning – in the wet, in the dry, on all tyres. The moment the car is unbalanced it slides all over the place, and then you overheat the tyres and nothing goes any more.
"It's a bit up and down with our car. We seem to be performing exceptionally good on some tracks - and then others where we have these moments where we are lacking the understanding why we are not performing."
Mercedes last won in Malaysia in 2014, with Ferrari and Red Bull claiming victory in 2015 and 2016 respectively.