He wants to know whether he should return to Scotland in order to try and regain his father's throne, or whether this will result in his being killed. To test this, he challenges Macduff. He wants to know whether Macduff will care about Malcolm being a terrible king, which he would if he was a loyal Scotsman.
Why does Malcolm test Macduff and his loyalty?
Malcolm tests Macduff's loyalty by pretending to be a worse human being than Macbeth himself. ... Macduff wants to rid his country of the tyrant Macbeth and vows to help Malcolm achieve this. When he learns of the murder of his family he becomes even more determined to take revenge.
Why does Malcolm question Macduff?
Why does Malcolm question Macduff's loyalty? Malcolm thinks Macduff may be working for Macbeth, because the two were once friends. ... He wants Malcolm and Macduff to return to Scotland and raise an army to oppose Macbeth.