Deepfake detection challenge from R

Image Recognition & Image Processing

A couple of months ago, Amazon, Facebook, Microsoft, and other contributors initiated a challenge consisting of telling apart real and AI-generated (“fake”) videos. We show how to approach this challenge from R.

Turgut Abdullayev https://github.com/henry090 (QSS Analytics)http://www.qss.az/
08-18-2020

Introduction

Working with video datasets, particularly with respect to detection of AI-based fake objects, is very challenging due to proper frame selection and face detection. To approach this challenge from R, one can make use of capabilities offered by OpenCV, magick, and keras.

Our approach consists of the following consequent steps:

Let’s quickly introduce the non-deep-learning libraries we’re using. OpenCV is a computer vision library that includes:

On the other hand, magick is the open-source image-processing library that will help to read and extract useful features from video datasets:

Before we go into a detailed explanation, readers should know that there is no need to copy-paste code chunks. Because at the end of the post one can find a link to Google Colab with GPU acceleration. This kernel allows everyone to run and reproduce the same results.

Data exploration

The dataset that we are going to analyze is provided by AWS, Facebook, Microsoft, the Partnership on AI’s Media Integrity Steering Committee, and various academics.

It contains both real and AI-generated fake videos. The total size is over 470 GB. However, the sample 4 GB dataset is separately available.

Frame extraction

The videos in the folders are in the format of mp4 and have various lengths. Our task is to determine the number of images to capture per second of a video. We usually took 1-3 fps for every video.

Note: Set fps to NULL if you want to extract all frames.

video = magick::image_read_video("aagfhgtpmv.mp4",fps = 2)
vid_1 = video[[1]]
vid_1 = magick::image_read(vid_1) %>% image_resize('1000x1000')

We saw just the first frame. What about the rest of them?

Looking at the gif one can observe that some fakes are very easy to differentiate, but a small fraction looks pretty realistic. This is another challenge during data preparation.

Face detection

At first, face locations need to be determined via bounding boxes, using OpenCV. Then, magick is used to automatically extract them from all images.

# get face location and calculate bounding box
library(opencv)
unconf <- ocv_read('frame_1.jpg')
faces <- ocv_face(unconf)
facemask <- ocv_facemask(unconf)
df = attr(facemask, 'faces')
rectX = (df$x - df$radius) 
rectY = (df$y - df$radius)
x = (df$x + df$radius) 
y = (df$y + df$radius)

# draw with red dashed line the box
imh  = image_draw(image_read('frame_1.jpg'))
rect(rectX, rectY, x, y, border = "red", 
     lty = "dashed", lwd = 2)
dev.off()

Face extraction

If face locations are found, then it is very easy to extract them all.

edited = image_crop(imh, "49x49+66+34")
edited = image_crop(imh, paste(x-rectX+1,'x',x-rectX+1,'+',rectX, '+',rectY,sep = ''))
edited

Deep learning model

After dataset preparation, it is time to build a deep learning model with Keras. We can quickly place all the images into folders and, using image generators, feed faces to a pre-trained Keras model.

train_dir = 'fakes_reals'
width = 150L
height = 150L
epochs = 10

train_datagen = image_data_generator(
  rescale = 1/255,
  rotation_range = 40,
  width_shift_range = 0.2,
  height_shift_range = 0.2,
  shear_range = 0.2,
  zoom_range = 0.2,
  horizontal_flip = TRUE,
  fill_mode = "nearest",
  validation_split=0.2
)


train_generator <- flow_images_from_directory(
  train_dir,                  
  train_datagen,             
  target_size = c(width,height), 
  batch_size = 10,
  class_mode = "binary"
)

# Build the model ---------------------------------------------------------

conv_base <- application_vgg16(
  weights = "imagenet",
  include_top = FALSE,
  input_shape = c(width, height, 3)
)

model <- keras_model_sequential() %>% 
  conv_base %>% 
  layer_flatten() %>% 
  layer_dense(units = 256, activation = "relu") %>% 
  layer_dense(units = 1, activation = "sigmoid")

model %>% compile(
  loss = "binary_crossentropy",
  optimizer = optimizer_rmsprop(lr = 2e-5),
  metrics = c("accuracy")
)

history <- model %>% fit_generator(
  train_generator,
  steps_per_epoch = ceiling(train_generator$samples/train_generator$batch_size),
  epochs = 10
)



Reproduce in a Notebook

Conclusion

This post shows how to do video classification from R. The steps were:

However, readers should know that the implementation of the following steps may drastically improve model performance:

Feel free to try these options on the Deepfake detection challenge and share your results in the comments section!

Thanks for reading!

Corrections

If you see mistakes or want to suggest changes, please create an issue on the source repository.

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. Source code is available at https://github.com/henry090/Deepfake-from-R, unless otherwise noted. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

Citation

For attribution, please cite this work as

Abdullayev (2020, Aug. 18). Posit AI Blog: Deepfake detection challenge from R. Retrieved from https://blogs.rstudio.com/tensorflow/posts/2020-08-18-deepfake/

BibTeX citation

@misc{abdullayev2020deepfake,
  author = {Abdullayev, Turgut},
  title = {Posit AI Blog: Deepfake detection challenge from R},
  url = {https://blogs.rstudio.com/tensorflow/posts/2020-08-18-deepfake/},
  year = {2020}
}